Test-retest reliability for common tasks in vision science

Are we capturing individual differences? Evaluating the test-retest reliability of experimental tasks used to measure social cognitive abilities

Social cognitive skills are crucial for positive interpersonal relationships, health, and wellbeing and encompass both automatic and reflexive processes. To assess this myriad of skills, researchers have developed numerous experimental tasks that measure automatic imitation, emotion recognition, empathy, perspective taking, and intergroup bias and have used these to reveal important individual differences in social cognition. However, the very reason these tasks produce robust experimental effects - low between-participant variability - can make their use as correlational tools problematic. We performed an evaluation of test-retest reliability for common experimental tasks that measure social cognition. One-hundred and fifty participants completed the race-Implicit Association Test (r-IAT), Stimulus-Response Compatibility (SRC) task, Emotional Go/No-Go (eGNG) task, Dot Perspective-Taking (DPT) task, and State Affective Empathy (SAE) task, as well as the Interpersonal Reactivity Index (IRI) and indices of Explicit Bias (EB) across two sessions within 3 weeks. Estimates of test-retest reliability varied considerably between tasks and their indices: the eGNG task had good reliability (ICC = 0.63-0.69); the SAE task had moderate-to-good reliability (ICC = 0.56-0.77); the r-IAT had moderate reliability (ICC = 0.49); the DPT task had poor-to-good reliability (ICC = 0.24-0.60); and the SRC task had poor reliability (ICC = 0.09-0.29). The IRI had good-to-excellent reliability (ICC = 0.76-0.83) and EB had good reliability (ICC = 0.70-0.77). Experimental tasks of social cognition are used routinely to assess individual differences, but their suitability for this is rarely evaluated. Researchers investigating individual differences must assess the test-retest reliability of their measures.

Quantifying sighting dominance using on-display projections of monocular and binocular views

Sighting dominance is an important behavioral property which has been difficult to measure quantitatively with high precision. We developed a measurement method that is grounded in a two-camera model that satisfies these aims. Using a simple alignment task, this method quantifies sighting ocular dominance during binocular viewing, identifying each eye's relative contribution to binocular vision. The method involves placing a physical target between the viewer and a display. The viewer indicates the perceived target's projection on the display with both eyes open and with only one eye open. The relative location of the binocular projection in relation to the two monocular projections is the index of dominance. The method produces a continuous variable with robust test-retest reliability (ICC = 0.96). The unit of measurement for the computed quantity is physiologically grounded: it is proportional to the distance between the monocular projections, which we show is predictable from interpupillary distance and phoria. Comparisons with the classic 'hole in card' sighting dominance test show good agreement, but also hint at potential bias for determining right-eye dominance in the latter. Interestingly, we find that some individuals systematically demonstrate nearly balanced vision, a phenomenon previously construed as mixed dominance or noisy responses. We also present ways to quantify and mitigate sources of random noise in this measurement. Overall, this new method allows for precise estimation of sighting dominance during binocular viewing. We expect it will allow a more effective understanding of the neural basis of dominance and improved effectiveness when using sighting dominance as a covariate in more complex analyses.

A multiverse assessment of the reliability of the self-matching task as a measurement of the self-prioritization effect

The self-matching task (SMT) is widely used to investigate the cognitive mechanisms underlying the self-prioritization effect (SPE), wherein performance is enhanced for self-associated stimuli compared to other-associated ones. Although the SMT robustly elicits the SPE, there is a lack of data quantifying the reliability of this paradigm. This is problematic, given the prevalence of the reliability paradox in cognitive tasks: many well-established cognitive tasks demonstrate relatively low reliability when used to evaluate individual differences, despite exhibiting replicable effects at the group level. To fill this gap, this preregistered study investigated the reliability of SPE derived from the SMT using a multiverse approach, combining all possible indicators and baselines reported in the literature. We first examined the robustness of 24 SPE measures across 42 datasets (N = 2250) using a meta-analytical approach. We then calculated the split-half reliability (r) and intraclass correlation coefficient (ICC2) for each SPE measure. Our findings revealed a robust group-level SPE across datasets. However, when evaluating individual differences, SPE indices derived from reaction time (RT) and efficiency exhibited relatively higher, compared to other SPE indices, but still unsatisfied split-half reliability (approximately 0.5). The reliability across multiple time points, as assessed by ICC2, RT, and efficiency, demonstrated moderate levels of test-retest reliability (close to 0.5). These findings revealed the presence of a reliability paradox in the context of SMT-based SPE assessment. We discussed the implications of how to enhance individual-level reliability using this paradigm for future study design.

Reliably measuring learning-dependent distractor suppression with eye tracking

In the field of psychological science, behavioral performance in computer-based cognitive tasks often exhibits poor reliability. The absence of reliable measures of cognitive processes contributes to non-reproducibility in the field and impedes the investigation of individual differences. Specifically in visual search paradigms, response time-based measures have shown poor test-retest reliability and internal consistency across attention capture and distractor suppression, but one study has demonstrated the potential for oculomotor measures to exhibit superior reliability. Therefore, in this study, we investigated three datasets to compare the reliability of learning-dependent distractor suppression measured via distractor fixations (oculomotor capture) and latency to fixate the target (fixation times). Our findings reveal superior split-half reliability of oculomotor capture compared to that of fixation times regardless of the critical distractor comparison, with the reliability of oculomotor capture in most cases falling within the range that is acceptable for the investigation of individual differences. We additionally find that older adults have superior oculomotor reliability compared with young adults, potentially addressing a significant limitation in the aging literature of high variability in response time measures due to slower responses. Our findings highlight the utility of measuring eye movements in the pursuit of reliable indicators of distractor processing and the need to further test and develop additional measures in other sensory domains to maximize statistical power, reliability, and reproducibility.

Individual differences in visual search: A systematic review of the link between visual search performance and traits or abilities

Visual search (VS) comprises a class of tasks that we typically perform several times during a day and requires intentionally scanning (with or without moving the eyes) the environment for a specific target (be it an object or a feature) among distractor stimuli. Experimental research in lab-based or real-world settings has offered insight into its underlying neurocognitive mechanisms from a nomothetic point of view. A lesser-known but rapidly growing body of quasi-experimental and correlational research has explored the link between individual differences and VS performance. This combines different research traditions and covers a wide range of individual differences in studies deploying a vast array of VS tasks. As such, it is a challenge to determine whether any associations highlighted in single studies are robust when considering the wider literature. However, clarifying such relationships systematically and comprehensively would help build more accurate models of VS, and it would highlight promising directions for future research. This systematic review provides an up to date and comprehensive synthesis of the existing literature investigating associations between common indices of performance in VS tasks and measures of individual differences mapped onto four categories of cognitive abilities (short-term working memory, fluid reasoning, visual processing and processing speed) and seven categories of traits (Big Five traits, trait anxiety and autistic traits). Consistent associations for both traits (in particular, conscientiousness, autistic traits and trait anxiety - the latter limited to emotional stimuli) and cognitive abilities (particularly visual processing) were identified. Overall, however, informativeness of future studies would benefit from checking and reporting the reliability of all measurement tools, applying multiplicity correction, using complementary techniques, study preregistration and testing why, rather than only if, a robust relation between certain individual differences and VS performance exists.

The relationship between cognitive and affective control and adolescent mental health

Background

Cognitive control problems have been implicated in the etiology and maintenance of mental health problems, including depression, in adults. Studies in adolescents have been more equivocal, with some showing changes in cognitive control in adolescents with mental health problems, whereas others fail to show an association. This study examines whether adolescent mental health is associated with affective control, the application of cognitive control in affective contexts, which shows more protracted development than cognitive control.

Methods

The present study investigated the association of cognitive and affective control with depressive symptomatology and self-reported diagnostic history of mental health problems in adolescents. The study included 1929 participants (M age = 13.89) from the Future Proofing Study (N = 6,388, 11-16 years), who completed affective (incl., affective stimuli) and/or cognitive (incl., neutral stimuli) versions of a working memory (backward digit-span) and/or shifting (card-sorting) task at least once within 3 weeks of assessing mental health.

Results

Poorer working memory was associated with greater depressive symptomatology in adolescents (β = -0.06, p = .004), similarly across cognitive and affective control conditions (β = -0.02, p = .269). Adolescents with self-reported diagnostic history of mental health problems had significantly poorer shifting ability in affective compared to cognitive control conditions (b = 0.05, p = .010), whereas for adolescents with no self-reported diagnoses, shifting ability did not differ between conditions (b = -0.00, p = .649).

Conclusions

The present analyses suggest that working memory difficulties, in particular, may be associated with the experience of current depressed mood in adolescents. Problems with affective shifting may be implicated in a range of mental health problems in adolescents. Given the ubiquitous need for efficient cognitive functioning in daily life, enhancing cognitive and affective control in adolescents may be a promising means of improving functioning across a range of domains, including affective functioning, and by extension, adolescent mental health.

Weak correlations between visual abilities in healthy older adults, despite long-term performance stability

Using batteries of visual tests, most studies have found that there are only weak correlations between the performance levels of the tests. Factor analysis has confirmed these results. This means that a participant excelling in one test may rank low in another test. Hence, there is very little evidence for a common factor in vision. In visual aging research, cross-sectional studies have repeatedly found that healthy older adults' performance is strongly deteriorated in most visual tests compared to young adults. However, also within the healthy older population, there is no evidence for a visual common factor. To investigate whether the weak between-tests correlations are due to fluctuations in individual performance throughout time, we conducted a longitudinal study. Healthy older adults performed a battery of eight visual tests, with two re-tests after approximately four and seven years. Pearson's, Spearman's and intraclass correlations of most visual tests were significant across the three testing, indicating that the tests are reliable and individual differences are stable across years. Yet, we found low between-tests correlations at each visit, which is consistent with previous studies finding no evidence for a visual common factor. Our results exclude the possibility that the weak correlations between tests are due to high within-individual variance across time.

Translation and validation of the Polish language version of the Teenagers Quality of Life questionnaire (T-QoL)

Introduction

Skin diseases affect patients at any age, but as each period in life is different, tools used to assess quality of life impairment should be adjusted according to the particular age group. Adolescence is a unique time, when young individuals go through many changes, making them especially vulnerable to stress.

Aim

Translation and validation of a Polish language version of the Teenagers Quality of Life questionnaire (T-QoL) questionnaire.

Material and methods

T-QoL was translated following international guidelines. A group of 34 dermatological patients, aged 12-19 years old, with various skin diseases were given the T-QoL as well as the CDLQI or DLQI to complete. They were also asked to complete the T-QoL questionnaire for the second time after 3-5 days. Statistical analysis of the results was performed.

Results

The Polish version of T-QoL is internally consistent (Cronbach α 0.893 for the whole questionnaire). Moreover, it presents very good convergent validity (ICC = 0.864). No statistically significant differences between each question were noticed between the first and second time of completing the form. T-QoL scores correlated significantly with DLQI (p = 0.008, r = 0.636) and CDLQI (p < 0.001, r = 0.777) scores.

Conclusions

The Polish version of the T-QoL questionnaire is a reliable instrument with adequate convergent validity, consistency and reproducibility. It can be successfully used to measure quality of life impairment among teenagers.

Exploring the extent to which shared mechanisms contribute to motion-position illusions

Motion-position illusions (MPIs) are visual motion illusions in which motion signals bias the perceived position of an object. Due to phenomenological similarities between these illusions, previous research has assumed that some are caused by common mechanisms. However, this assumption has yet to be directly tested. This study investigates this assumption by exploiting between-participant variations in illusion magnitude. During two sessions, 106 participants viewed the flash-lag effect, luminance flash-lag effect, Fröhlich effect, flash-drag effect, flash-grab effect, motion-induced position shift, twinkle-goes effect, and the flash-jump effect. For each effect, the magnitude of the illusion was reliable within participants, strongly correlating between sessions. When the pairwise correlations of averaged illusions magnitudes were explored, two clusters of statistically significant positively correlated illusions were identified. The first cluster comprised the flash-grab effect, motion-induced position shift, and twinkle-goes effect. The second cluster comprised the Fröhlich and flash-drag effect. The fact that within each of these two clusters, individual differences in illusion magnitude were correlated suggests that these clusters may reflect shared underlying mechanisms. An exploratory factor analysis provided additional evidence that these correlated clusters shared an underlying factor, with each cluster loading onto their own factor. Overall, our results reveal that, contrary to the prevailing perspective in the literature, while some motion-position illusions share processes, most of these illusions are unlikely to reflect any shared processes, instead implicating unique mechanisms.

Test-retest reliability of eye tracking measures in a computerized Trail Making Test

The Trail Making Test (TMT) is a frequently applied neuropsychological test that evaluates participants' executive functions based on their time to connect a sequence of numbers (TMT-A) or alternating numbers and letters (TMT-B). Test performance is associated with various cognitive functions ranging from visuomotor speed to working memory capabilities. However, although the test can screen for impaired executive functioning in a variety of neuropsychiatric disorders, it provides only little information about which specific cognitive impairments underlie performance detriments. To resolve this lack of specificity, recent cognitive research combined the TMT with eye tracking so that eye movements could help uncover reasons for performance impairments. However, using eye-tracking-based test scores to examine differences between persons, and ultimately apply the scores for diagnostics, presupposes that the reliability of the scores is established. Therefore, we investigated the test-retest reliabilities of scores in an eye-tracking version of the TMT recently introduced by Recker et al. (2022). We examined two healthy samples performing an initial test and then a retest 3 days (n = 31) or 10 to 30 days (n = 34) later. Results reveal that, although reliabilities of classic completion times were overall good, comparable with earlier versions, reliabilities of eye-tracking-based scores ranged from excellent (e.g., durations of fixations) to poor (e.g., number of fixations guiding manual responses). These findings indicate that some eye-tracking measures offer a strong basis for assessing interindividual differences beyond classic behavioral measures when examining processes related to information accumulation processes but are less suitable to diagnose differences in eye-hand coordination.

Seeing Through Each Other's Hearts: Inferring Others' Heart Rate as a Function of Own Heart Rate Perception and Perceived Social Intelligence

Successful social interactions require a good understanding of the emotional states of other people. This information is often not directly communicated but must be inferred. As all emotional experiences are also imbedded in the visceral or interoceptive state of the body (i.e., accelerating heart rate during arousal), successfully inferring the interoceptive states of others may open a window into their emotional state. But how well can people do that? Here, we replicate recent results showing that people can discriminate between the cardiac states (i.e., the resting heartrate) of other people by simply looking at them. We further tested whether the ability to infer the interoceptive states of others depends on one's own interoceptive abilities. We measured people's performance in a cardioception task and their self-reported interoceptive accuracy. Whilst neither was directly associated to their ability to infer the heartrate of another person, we found a significant interaction. Specifically, overestimating one's own interoceptive capacities was associated with a worse performance at inferring the heartrate of others. In contrast, underestimating one's own interoceptive capacities did not have such influence. This pattern suggests that deficient beliefs about own interoceptive capacities can have detrimental effects on inferring the interoceptive states of other people.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-022-00151-4.