Are the attention checks embedded in delay discounting tasks a valid marker for data quality?

From error to insight: Removing non-systematic responding data in the delay discounting task may introduce systematic bias

Delay discounting (DD), which reflects a tendency to devalue rewards as the time to their receipt increases, is associated with health behaviors such as sleep disturbances, obesity, and externalizing behavior among adolescents. Response patterns characterized by inconsistent or unexpected reward valuation, called non-systematic responding (NSR), may also predict health outcomes. Many researchers flag and exclude NSR trials prior to analysis, which could lead to systematic bias if NSR (a) varies by demographic characteristics or (b) predicts health outcomes. Thus, in this study we characterized NSR and examined its potential beyond error by comparing it against DD with a secondary data analysis of the Adolescent Brain Cognitive Development (ABCD) Study-a population-based study that tracked youths (N = 11,948) annually from 8 to 11 years of age over 4 years. We assessed DD and NSR using the Adjusting Delay Discounting Task when youths were approximately 9.48 years old (SD = 0.51). We also examined three maladaptive health outcomes annually: sleep disturbances, obesity, and externalizing psychopathology. Our analysis revealed variations in NSR across races, ethnicities, and body mass index categories, with no significant differences observed by sex or gender. Notably, NSR was a stronger predictor of obesity and externalizing psychopathology than DD and inversely predicted the growth trajectory of obesity. These findings suggest that removing NSR patterns could systematically bias analyses given that NSR may capture unexplored response variability. This study demonstrates the significance of NSR and underscores the necessity for further research on how to manage NSR in future DD studies.

Delay discounting data in the Adolescent Brain Cognitive Development (ABCD) study: Modeling and analysis considerations

This report provides a primer to delay discounting data in the context of the Adolescent Brain Cognitive Development (ABCD) Study. Delay discounting describes the tendency for organisms to devalue temporally constrained outcomes. This decision-making framework has garnered attention from multiple fields for its association with various behavioral health conditions like substance use disorder. Importantly, the literature on delay discounting describes many approaches to analyzing and interpreting discounting data. To be most beneficial to the broader scientific audience, consistency and reproducibility in how delay discounting data are operationalized, analyzed, and interpreted is key. We describe relevant data analysis methods for use with the ABCD Study, a large-cohort longitudinal study (N = 11,878) examining delay discounting among youth respondents across child and adolescent development. Particular attention is given to data collected from children and younger populations given their relevance to ABCD research and potential merit for unique analytic considerations (e.g., higher rates of atypical responding). We first provide a background on the broad theoretical and conceptual aspects of discounting research. We then review discounting assessment, describing conventional titration tasks and the more novel algorithm-based approaches to generating descriptive metrics. We conclude with recommendations for best practice modeling, data handling and exclusions based on nonsystematic data, and ensuing interpretations. Analytic pipelines and coding are provided for investigator use. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

Naturalistic substance use before/during MTurk research participation is associated with increased substance demand and craving

Although crowdsourcing platforms are widely used in substance-use research, it is unclear what percentage of participants use substances at the time of participation and how this might affect data quality, behavioral outcomes, or decision making. We conducted a secondary analysis of data collected on MTurk for a two-session, within-subject experiment recruiting individuals who regularly use alcohol, cannabis, cigarettes, or opioids. We analyzed 527 observations collected across two sessions (Session 1: n = 303, Session 2: n = 224) on measures of substance use before (within 3 hr)/during participation, data quality, demand in hypothetical purchase tasks, delay discounting, and craving. Substance use before/during participation was common (35.7%). Some participants reported substance use before/during both (25.4%) or only one (20.1%) of the sessions. Between-subject analyses of the first session data revealed that participants who used substances before/during participation did not differ on quality measures yet were slower to complete the survey. Controlling for individual differences in demographic variables and typical substance use, using a substance before/during participation was associated with increased hypothetical consumption of substances when the substance was free (demand intensity) and higher craving for substances, but not delay discounting. Substance use before/during MTurk participation among individuals who regularly use substances is prevalent and may impact outcome measures or standardization across sessions in repeated measures designs. Several implications have emerged, including statistically or experimentally controlling for substance use occurring before/during participation, which could improve the validity and rigor of online substance use research, and should be considered a part of best practices. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

High in the Cloud: Alcohol-, Cannabis-, and Co-Use Before and During Remote Research Participation

OBJECTIVE

The use of crowdsourcing for addiction research has increased exponentially in recent years, but the extent to which the populations we expect results to generalize to might be engaging in substance use while participating in remote research has not been formally quantified. Understanding rates of day-of-study substance use on crowdsourcing platforms may be especially relevant given immediately recent use can alter cognitive and behavioral decision-making processes (e.g., attention, behavioral economic drug purchase tasks) that are often the focus of online substance use research.

METHOD

The purpose of this study is to (1) characterize rates of substance use (i.e., alcohol, cannabis, or both) among 790 Prolific workers on the day of the study, within the past three hours, and since starting the study; (2) provide sample demographic descriptive statistics, typical substance use patterns, and their associations with day-of use; and (3) evaluate whether day-of use is associated with time taken to complete the study and performance on attention checks.

RESULTS

Day-of use was greater than 10%, primarily consisted of cannabis use, and several general use patterns were associated with day-of use (e.g., past year binge drinking was associated with day-of cannabis use). Day-of use was not associated with longer study completion times; attention check analyses were inconclusive.

CONCLUSION

Considering these results, we provide suggestions for best practices when crowdsourcing data for addiction research and advocate for future studies that use naturalistic experiments to complement laboratory drug- and alcohol-administration studies.

Are the attention checks embedded in delay discounting tasks a valid marker for data quality?

To ensure good quality delay discounting (DD) data in research recruiting via crowdsourcing platforms, including attention checks within DD tasks have become common. These attention checks are typically identical in format to the task questions but have one sensical answer (e.g., "Would you prefer $0 now or $100 in a month?"). However, the validity of these attention checks as a marker for DD or overall survey data quality has not been directly examined. To address this gap, using data from two studies (total N = 700), the validity of these DD attention checks was tested by assessing performance on other non-DD attention checks and data quality measures both specific to DD and overall survey data (e.g., providing nonsystematic DD data, responding inconsistently in questionnaires). We also tested whether failing the attention checks was associated with degree of discounting or other participant characteristics to screen for potential bias. While failing the DD attention checks was associated with a greater likelihood of nonsystematic DD data, their discriminability was inadequate, and failure was sometimes associated with individual differences (suggesting that data exclusion might introduce bias). Failing the DD attention checks was also not associated with failing other attention checks or data quality indicators. Overall, the DD attention checks do not appear to be an adequate indicator of data quality on their own, for either the DD task or surveys overall. Strategies to enhance the validity of DD attention checks and data cleaning procedures are suggested, which should be evaluated in future research. (PsycInfo Database Record (c) 2023 APA, all rights reserved).