Inferring Cognitive Abilities from Response Times to Web-Administered Survey Items in a Population-Representative Sample

The added value of metadata on test completion time for the quantification of cognitive functioning in survey research

OBJECTIVE

Information on the time spent completing cognitive testing is often collected, but such data are not typically considered when quantifying cognition in large-scale community-based surveys. We sought to evaluate the added value of timing data over and above traditional cognitive scores for the measurement of cognition in older adults.

METHOD

We used data from the Longitudinal Aging Study in India-Diagnostic Assessment of Dementia (LASI-DAD) study (N = 4,091), to assess the added value of timing data over and above traditional cognitive scores, using item-specific regression models for 36 cognitive test items. Models were adjusted for age, gender, interviewer, and item score.

RESULTS

Compared to Quintile 3 (median time), taking longer to complete specific items was associated (p < 0.05) with lower cognitive performance for 67% (Quintile 5) and 28% (Quintile 4) of items. Responding quickly (Quintile 1) was associated with higher cognitive performance for 25% of simpler items (e.g., orientation for year), but with lower cognitive functioning for 63% of items requiring higher-order processing (e.g., digit span test). Results were consistent in a range of different analyses adjusting for factors including education, hearing impairment, and language of administration and in models using splines rather than quintiles.

CONCLUSIONS

Response times from cognitive testing may contain important information on cognition not captured in traditional scoring. Incorporation of this information has the potential to improve existing estimates of cognitive functioning.

The Understanding America Study (UAS)

PURPOSE

The Understanding America Study (UAS) is a probability-based Internet panel housed at the Center for Economic and Social Research at the University of Southern California (USC). The UAS serves as a social and health sciences infrastructure for collecting data on the daily lives of US families and individuals. The collected information includes survey data, DNA from saliva samples, information from wearables, contextual and administrative linkages, ecological momentary assessments, self-recorded narratives and electronic records of financial transactions. The information collected focuses on a defining challenge of our time-identifying factors explaining racial, ethnic, geographic and socioeconomic disparities over the life course, including racial discrimination, inequalities in access to education and healthcare, differences in physical, economic and social environments, and, more generally, the various opportunities and obstacles one encounters over the life course. The UAS infrastructure aims to optimise engagement with the wider research community both in data dissemination and in soliciting input on content and methods. To encourage input from the research community, we have reserved 100 000 min of survey time per year for outside researchers, who can propose to add survey questions four times a year.

PARTICIPANTS

The UAS currently comprises about 15 000 US residents (including a 3500-person California oversample) recruited by Address-Based Sampling and provided with Internet-enabled tablets if needed. Surveys are conducted in English and Spanish.

FINDINGS TO DATE

Since the founding of the UAS in 2014, we have conducted more than 600 surveys, including a sequence of surveys collecting biennial information on health and retirement (the complete Health and Retirement Study instrument), 11 cognitive assessments, personality, knowledge and use of information on Social Security programme rules, work disability and subjective well-being. Several hundreds of papers have been published based on the collected data in the UAS. Studies include documentations of the mental health effects of the COVID-19 pandemic and how this varied across socioeconomic groups; comparisons of physical activity measured with accelerometers and by self-reports showing the dramatic biases in the latter; extensive studies have shown the power of using paradata in gauging cognitive change over time; several messaging experiments have shown the effectiveness of information provision on the quality of decision-making affecting well-being at older ages.

FUTURE PLANS

The UAS national sample is planned to grow to 20 000 respondents by 2025, with subsamples of about 2500 African American, 2000 Asian and 3000 Hispanic participants and an oversample of rural areas. An increasing amount of non-interview data (contextual information, data from a suite of wearables and administrative linkages) is continually being added to the data files.

Can you tell people's cognitive ability level from their response patterns in questionnaires?

Questionnaires are ever present in survey research. In this study, we examined whether an indirect indicator of general cognitive ability could be developed based on response patterns in questionnaires. We drew on two established phenomena characterizing connections between cognitive ability and people's performance on basic cognitive tasks, and examined whether they apply to questionnaires responses. (1) The worst performance rule (WPR) states that people's worst performance on multiple sequential tasks is more indicative of their cognitive ability than their average or best performance. (2) The task complexity hypothesis (TCH) suggests that relationships between cognitive ability and performance increase with task complexity. We conceptualized items of a questionnaire as a series of cognitively demanding tasks. A graded response model was used to estimate respondents' performance for each item based on the difference between the observed and model-predicted response ("response error" scores). Analyzing data from 102 items (21 questionnaires) collected from a large-scale nationally representative sample of people aged 50+ years, we found robust associations of cognitive ability with a person's largest but not with their smallest response error scores (supporting the WPR), and stronger associations of cognitive ability with response errors for more complex than for less complex questions (supporting the TCH). Results replicated across two independent samples and six assessment waves. A latent variable of response errors estimated for the most complex items correlated .50 with a latent cognitive ability factor, suggesting that response patterns can be utilized to extract a rough indicator of general cognitive ability in survey research.

Visual-Motor Integration (VMI) Is Also Relevant for Computer, Smartphone, and Tablet Use by Adults: Introducing the Brief Box Clicking Test

IMPORTANCE

Visual-motor integration (VMI) is typically examined in children to promote handwriting, but it may also be relevant for adults' capacity for technology use.

OBJECTIVE

To examine the reliability and validity of speed of completion of the box clicking test, a web-based test of VMI.

DESIGN

Participants in the Understanding America Study completed online surveys on a regular basis, including a very brief (less than 30 s) self-administered box clicking test. For validity testing, we examined whether box clicking speed was associated with constructs relevant to visual-perceptual skills and motor coordination, the skills underlying VMI. Test-retest reliability was examined by computation of intraclass correlation coefficients.

PARTICIPANTS

A total of 11,114 adults.

MEASURES

Measures included the completion time for the box clicking task and measures relevant to visual perception (e.g., perceptual speed) and motor coordination (e.g., self-reported functional limitation).

RESULTS

Results suggested that the box clicking test was a VMI task. Slower test performance was associated with lower visual-perceptual speed and a greater likelihood of reporting difficulties with dressing, a motor coordination relevant task. Box clicking tests taken within at least 2 yr of one another had moderate test-retest stability, but future studies are needed to examine test-retest reliabilities over brief (e.g., 2-wk) time intervals.

CONCLUSIONS AND RELEVANCE

The box clicking test may serve both as a tool for research and to clinically observe whether clients have VMI difficulties that interfere with computer, smartphone, or tablet use. Plain-Language Summary: Use of devices such as smartphones and computers is increasingly becoming integral for daily functioning. Visual-motor integration (VMI) has often been addressed by occupational therapists to support handwriting of children, but it may also be important for technology use by adults. Prior literature supports the relevance of VMI to technology use, and adults with various chronic conditions have been found to have decrements in VMI. We tested the psychometric properties of a brief box clicking test of VMI that could be used to examine VMI underlying technology use among adults. Overall, results suggested that the box clicking test was a VMI task. Just as speed of gait has been used as an index of functional mobility, speed on the box clicking task seemed serviceable as an index of VMI ability. The box clicking test may also be used for clinical observation of whether VMI interferes with technology use.

Using Item Response Times in Online Questionnaires to Detect Mild Cognitive Impairment

OBJECTIVES

With the increase in web-based data collection, response times (RTs) for survey items have become a readily available byproduct in most online studies. We examined whether RTs in online questionnaires can prospectively discriminate between cognitively normal respondents and those with cognitive impairment, no dementia (CIND).

METHOD

Participants were 943 members of a nationally representative internet panel, aged 50 and older. We analyzed RTs that were passively recorded as paradata for 37 surveys (1,053 items) administered online over 6.5 years. A multilevel location-scale model derived 3 RT parameters for each survey: (1) a respondent's average RT and 2 components of intraindividual RT variability addressing (2) systematic RT adjustments and (3) unsystematic RT fluctuations. CIND status was determined at the end of the 6.5-year period.

RESULTS

All 3 RT parameters were significantly associated with CIND, with a combined predictive accuracy of area under the receiver-operating characteristic curve = 0.74. Slower average RTs, smaller systematic RT adjustments, and greater unsystematic RT fluctuations prospectively predicted a greater likelihood of CIND over periods of up to 6.5, 4.5, and 1.5 years, respectively.

DISCUSSION

RTs for survey items are a potential early indicator of CIND, which may enhance analyses of predictors, correlates, and consequences of cognitive impairment in online survey research.