Measurement error correction and sensitivity analysis in longitudinal dietary intervention studies using an external validation study

Prediction of 24-Hour Urinary Sodium Excretion Using Machine-Learning Algorithms

BACKGROUND

Accurate quantification of sodium intake based on self-reported dietary assessments has been a persistent challenge. We aimed to apply machine-learning (ML) algorithms to predict 24-hour urinary sodium excretion from self-reported questionnaire information.

METHODS AND RESULTS

We analyzed 3454 participants from the NHS (Nurses' Health Study), NHS-II (Nurses' Health Study II), and HPFS (Health Professionals Follow-Up Study), with repeated measures of 24-hour urinary sodium excretion over 1 year. We used an ensemble approach to predict averaged 24-hour urinary sodium excretion using 36 characteristics. The TOHP-I (Trial of Hypertension Prevention I) was used for the external validation. The final ML algorithms were applied to 167 920 nonhypertensive adults with 30-year follow-up to estimate confounder-adjusted hazard ratio (HR) of incident hypertension for predicted sodium. Averaged 24-hour urinary sodium excretion was better predicted and calibrated with ML compared with the food frequency questionnaire (Spearman correlation coefficient, 0.51 [95% CI, 0.49-0.54] with ML; 0.19 [95% CI, 0.16-0.23] with the food frequency questionnaire; 0.46 [95% CI, 0.42-0.50] in the TOHP-I). However, the prediction heavily depended on body size, and the prediction of energy-adjusted 24-hour sodium excretion was modestly better using ML. ML-predicted sodium was modestly more strongly associated than food frequency questionnaire-based sodium in the NHS-II (HR comparing Q5 versus Q1, 1.48 [95% CI, 1.40-1.56] with ML; 1.04 [95% CI, 0.99-1.08] with the food frequency questionnaire), but no material differences were observed in the NHS or HPFS.

CONCLUSIONS

The present ML algorithm improved prediction of participants' absolute 24-hour urinary sodium excretion. The present algorithms may be a generalizable approach for predicting absolute sodium intake but do not substantially reduce the bias stemming from measurement error in disease associations.

The Measurement Error Elephant in the Room: Challenges and Solutions to Measurement Error in Epidemiology

Measurement error, although ubiquitous, is uncommonly acknowledged and rarely assessed or corrected in epidemiologic studies. This review offers a straightforward guide to common problems caused by measurement error in research studies and a review of several accessible bias-correction methods for epidemiologists and data analysts. Although most correction methods require criterion validation including a gold standard, there are also ways to evaluate the impact of measurement error and potentially correct for it without such data. Technical difficulty ranges from simple algebra to more complex algorithms that require expertise, fine tuning, and computational power. However, at all skill levels, software packages and methods are available and can be used to understand the threat to inferences that arises from imperfect measurements.

Effects of differential measurement error in self-reported diet in longitudinal lifestyle intervention studies

Background

Lifestyle intervention studies often use self-reported measures of diet as an outcome variable to measure changes in dietary intake. The presence of measurement error in self-reported diet due to participant failure to accurately report their diet is well known. Less familiar to researchers is differential measurement error, where the nature of measurement error differs by treatment group and/or time. Differential measurement error is often present in intervention studies and can result in biased estimates of the treatment effect and reduced power to detect treatment effects. Investigators need to be aware of the impact of differential measurement error when designing intervention studies that use self-reported measures.

Methods

We use simulation to assess the consequences of differential measurement error on the ability to estimate treatment effects in a two-arm randomized trial with two time points. We simulate data under a variety of scenarios, focusing on how different factors affect power to detect a treatment effect, bias of the treatment effect, and coverage of the 95% confidence interval of the treatment effect. Simulations use realistic scenarios based on data from the Trials of Hypertension Prevention Study. Simulated sample sizes ranged from 110-380 per group.

Results

Realistic differential measurement error seen in lifestyle intervention studies can require an increased sample size to achieve 80% power to detect a treatment effect and may result in a biased estimate of the treatment effect.

Conclusions

Investigators designing intervention studies that use self-reported measures should take differential measurement error into account by increasing their sample size, incorporating an internal validation study, and/or identifying statistical methods to correct for differential measurement error.

Calibrating validation samples when accounting for measurement error in intervention studies

Many lifestyle intervention trials depend on collecting self-reported outcomes, such as dietary intake, to assess the intervention's effectiveness. Self-reported outcomes are subject to measurement error, which impacts treatment effect estimation. External validation studies measure both self-reported outcomes and accompanying biomarkers, and can be used to account for measurement error. However, in order to account for measurement error using an external validation sample, an assumption must be made that the inferences are transportable from the validation sample to the intervention trial of interest. This assumption does not always hold. In this paper, we propose an approach that adjusts the validation sample to better resemble the trial sample, and we also formally investigate when bias due to poor transportability may arise. Lastly, we examine the performance of the methods using simulation, and illustrate them using PREMIER, a lifestyle intervention trial measuring self-reported sodium intake as an outcome, and OPEN, a validation study measuring both self-reported diet and urinary biomarkers.

Characterizing Measurement Error in Dietary Sodium in Longitudinal Intervention Studies

Background: Previous measurement error work that investigates the relationship between a nutritional biomarker and self-reported intake levels has typically been at a single time point, in a single treatment group, or with respect to basic patient demographics. Few studies have examined the measurement error structure in longitudinal randomized trials, and whether the error varies across time or group. This structure is crucial to understand, however, in order to correct for measurement error in self-reported outcomes and properly interpret the longitudinal effects of dietary interventions. Methods: Using two longitudinal randomized controlled trials with internal longitudinal validation data (urinary biomarkers and self-reported values), we examine the relationship between urinary sodium and self-reported sodium and whether this relationship changes as a function of time and/or treatment condition. We do this by building a mixed effects regression model, allowing for a flexible error variance-covariance structure, and testing all possible interactions between time, treatment condition, and self-reported intake. Results: Using a backward selection approach, we arrived at the same final model for both validation data sets. We found no evidence that measurement error changes as a function of self-reported sodium. However, we did find evidence that urinary sodium can differ by time or treatment condition even when conditioning on self-reported values. Conclusion: In longitudinal nutritional intervention trials it is possible that measurement error differs across time and treatment groups. It is important for researchers to consider this possibility and not just assume non-differential measurement error. Future studies should consider data collection strategies to account for the potential dynamic nature of measurement error, such as collecting internal validation data across time and treatment groups when possible.