Using Item Response Times in Online Questionnaires to Detect Mild Cognitive Impairment

The added value of metadata on test completion time for the quantification of cognitive functioning in survey research

OBJECTIVE

Information on the time spent completing cognitive testing is often collected, but such data are not typically considered when quantifying cognition in large-scale community-based surveys. We sought to evaluate the added value of timing data over and above traditional cognitive scores for the measurement of cognition in older adults.

METHOD

We used data from the Longitudinal Aging Study in India-Diagnostic Assessment of Dementia (LASI-DAD) study (N = 4,091), to assess the added value of timing data over and above traditional cognitive scores, using item-specific regression models for 36 cognitive test items. Models were adjusted for age, gender, interviewer, and item score.

RESULTS

Compared to Quintile 3 (median time), taking longer to complete specific items was associated (p < 0.05) with lower cognitive performance for 67% (Quintile 5) and 28% (Quintile 4) of items. Responding quickly (Quintile 1) was associated with higher cognitive performance for 25% of simpler items (e.g., orientation for year), but with lower cognitive functioning for 63% of items requiring higher-order processing (e.g., digit span test). Results were consistent in a range of different analyses adjusting for factors including education, hearing impairment, and language of administration and in models using splines rather than quintiles.

CONCLUSIONS

Response times from cognitive testing may contain important information on cognition not captured in traditional scoring. Incorporation of this information has the potential to improve existing estimates of cognitive functioning.

Early Identification of Cognitive Impairment in Community Environments Through Modeling Subtle Inconsistencies in Questionnaire Responses: Machine Learning Model Development and Validation

BACKGROUND

The underdiagnosis of cognitive impairment hinders timely intervention of dementia. Health professionals working in the community play a critical role in the early detection of cognitive impairment, yet still face several challenges such as a lack of suitable tools, necessary training, and potential stigmatization.

OBJECTIVE

This study explored a novel application integrating psychometric methods with data science techniques to model subtle inconsistencies in questionnaire response data for early identification of cognitive impairment in community environments.

METHODS

This study analyzed questionnaire response data from participants aged 50 years and older in the Health and Retirement Study (waves 8-9, n=12,942). Predictors included low-quality response indices generated using the graded response model from four brief questionnaires (optimism, hopelessness, purpose in life, and life satisfaction) assessing aspects of overall well-being, a focus of health professionals in communities. The primary and supplemental predicted outcomes were current cognitive impairment derived from a validated criterion and dementia or mortality in the next ten years. Seven predictive models were trained, and the performance of these models was evaluated and compared.

RESULTS

The multilayer perceptron exhibited the best performance in predicting current cognitive impairment. In the selected four questionnaires, the area under curve values for identifying current cognitive impairment ranged from 0.63 to 0.66 and was improved to 0.71 to 0.74 when combining the low-quality response indices with age and gender for prediction. We set the threshold for assessing cognitive impairment risk in the tool based on the ratio of underdiagnosis costs to overdiagnosis costs, and a ratio of 4 was used as the default choice. Furthermore, the tool outperformed the efficiency of age or health-based screening strategies for identifying individuals at high risk for cognitive impairment, particularly in the 50- to 59-year and 60- to 69-year age groups. The tool is available on a portal website for the public to access freely.

CONCLUSIONS

We developed a novel prediction tool that integrates psychometric methods with data science to facilitate "passive or backend" cognitive impairment assessments in community settings, aiming to promote early cognitive impairment detection. This tool simplifies the cognitive impairment assessment process, making it more adaptable and reducing burdens. Our approach also presents a new perspective for using questionnaire data: leveraging, rather than dismissing, low-quality data.

Normative study of SATURN: a digital, self-administered, open-source cognitive assessment tool for Italians aged 50-80

Introduction

This study aimed to establish normative data for the Self-Administered Tasks Uncovering Risk of Neurodegeneration (SATURN), a brief computer-based test for global cognitive assessment through accuracy and response times on tasks related to memory, attention, temporal orientation, visuo-constructional abilities, math (calculation), executive functions, and reading speed.

Methods

A sample of 323 Italian individuals with Montreal Cognitive Assessment (MoCA) equivalent score ≥1 (180 females; average age: 61.33 years; average education: 11.32 years), stratified by age, education, and sex, completed SATURN using PsychoPy, and a paper-and-pencil protocol consisting of Mini-Mental State Examination (MMSE) and MoCA. Data analyses included: (i) correlations between the total accuracy scores of SATURN and those of MMSE and MoCA; (ii) multiple regressions to determine the impact of sex, age, and education, along with the computation of adjusted scores; (iii) the calculation of inner and outer tolerance limits, equivalent scores, and the development of correction grids.

Results

The mean total time on tasks was 6.72 ± 3.24 min. Age and education significantly influence the SATURN total accuracy, while sex influences the total time on tasks. Specific sociodemographic characteristics influence subdomain accuracies and times on task differently. For the adjusted SATURN total score, the outer limit corresponds to 16.56 out of 29.00 (cut-off), while the inner limit is 18.57. SATURN significantly correlates with MMSE and MoCA.

Discussion

In conclusion, SATURN is the first open-source digital tool for initial cognitive assessment in Italy, showing potential for self-administration in primary care, and remote administration. Future studies need to assess its sensitivity and specificity in detecting pathological cognitive decline.

The Understanding America Study (UAS)

PURPOSE

The Understanding America Study (UAS) is a probability-based Internet panel housed at the Center for Economic and Social Research at the University of Southern California (USC). The UAS serves as a social and health sciences infrastructure for collecting data on the daily lives of US families and individuals. The collected information includes survey data, DNA from saliva samples, information from wearables, contextual and administrative linkages, ecological momentary assessments, self-recorded narratives and electronic records of financial transactions. The information collected focuses on a defining challenge of our time-identifying factors explaining racial, ethnic, geographic and socioeconomic disparities over the life course, including racial discrimination, inequalities in access to education and healthcare, differences in physical, economic and social environments, and, more generally, the various opportunities and obstacles one encounters over the life course. The UAS infrastructure aims to optimise engagement with the wider research community both in data dissemination and in soliciting input on content and methods. To encourage input from the research community, we have reserved 100 000 min of survey time per year for outside researchers, who can propose to add survey questions four times a year.

PARTICIPANTS

The UAS currently comprises about 15 000 US residents (including a 3500-person California oversample) recruited by Address-Based Sampling and provided with Internet-enabled tablets if needed. Surveys are conducted in English and Spanish.

FINDINGS TO DATE

Since the founding of the UAS in 2014, we have conducted more than 600 surveys, including a sequence of surveys collecting biennial information on health and retirement (the complete Health and Retirement Study instrument), 11 cognitive assessments, personality, knowledge and use of information on Social Security programme rules, work disability and subjective well-being. Several hundreds of papers have been published based on the collected data in the UAS. Studies include documentations of the mental health effects of the COVID-19 pandemic and how this varied across socioeconomic groups; comparisons of physical activity measured with accelerometers and by self-reports showing the dramatic biases in the latter; extensive studies have shown the power of using paradata in gauging cognitive change over time; several messaging experiments have shown the effectiveness of information provision on the quality of decision-making affecting well-being at older ages.

FUTURE PLANS

The UAS national sample is planned to grow to 20 000 respondents by 2025, with subsamples of about 2500 African American, 2000 Asian and 3000 Hispanic participants and an oversample of rural areas. An increasing amount of non-interview data (contextual information, data from a suite of wearables and administrative linkages) is continually being added to the data files.

Can you tell people's cognitive ability level from their response patterns in questionnaires?

Questionnaires are ever present in survey research. In this study, we examined whether an indirect indicator of general cognitive ability could be developed based on response patterns in questionnaires. We drew on two established phenomena characterizing connections between cognitive ability and people's performance on basic cognitive tasks, and examined whether they apply to questionnaires responses. (1) The worst performance rule (WPR) states that people's worst performance on multiple sequential tasks is more indicative of their cognitive ability than their average or best performance. (2) The task complexity hypothesis (TCH) suggests that relationships between cognitive ability and performance increase with task complexity. We conceptualized items of a questionnaire as a series of cognitively demanding tasks. A graded response model was used to estimate respondents' performance for each item based on the difference between the observed and model-predicted response ("response error" scores). Analyzing data from 102 items (21 questionnaires) collected from a large-scale nationally representative sample of people aged 50+ years, we found robust associations of cognitive ability with a person's largest but not with their smallest response error scores (supporting the WPR), and stronger associations of cognitive ability with response errors for more complex than for less complex questions (supporting the TCH). Results replicated across two independent samples and six assessment waves. A latent variable of response errors estimated for the most complex items correlated .50 with a latent cognitive ability factor, suggesting that response patterns can be utilized to extract a rough indicator of general cognitive ability in survey research.