Reducing the Bias of Norm Scores in Non-Representative Samples: Weighting as an Adjunct to Continuous Norming Methods

We investigated whether the accuracy of normed test scores derived from non-demographically representative samples can be improved by combining continuous norming methods with compensatory weighting of test results. To this end, we introduce Raking, a method from social sciences, to psychometrics. In a simulated reference population, we modeled a latent cognitive ability with a typical developmental gradient, along with three demographic variables that were correlated to varying degrees with the latent ability. We simulated five additional populations representing patterns of non-representativeness that might be encountered in the real world. We subsequently drew smaller normative samples from each population and used an one-parameter logistic Item Response Theory (IRT) model to generate simulated test results for each individual. Using these simulated data, we applied norming techniques, both with and without compensatory weighting. Weighting reduced the bias of the norm scores when the degree of non-representativeness was moderate, with only a small risk of generating new biases.

Combining multiple imputation with raking of weights: An efficient and robust approach in the setting of nearly true models

Multiple imputation (MI) provides us with efficient estimators in model-based methods for handling missing data under the true model. It is also well-understood that design-based estimators are robust methods that do not require accurately modeling the missing data; however, they can be inefficient. In any applied setting, it is difficult to know whether a missing data model may be good enough to win the bias-efficiency trade-off. Raking of weights is one approach that relies on constructing an auxiliary variable from data observed on the full cohort, which is then used to adjust the weights for the usual Horvitz-Thompson estimator. Computing the optimally efficient raking estimator requires evaluating the expectation of the efficient score given the full cohort data, which is generally infeasible. We demonstrate MI as a practical method to compute a raking estimator that will be optimal. We compare this estimator to common parametric and semi-parametric estimators, including standard MI. We show that while estimators, such as the semi-parametric maximum likelihood and MI estimator, obtain optimal performance under the true model, the proposed raking estimator utilizing MI maintains a better robustness-efficiency trade-off even under mild model misspecification. We also show that the standard raking estimator, without MI, is often competitive with the optimal raking estimator. We demonstrate these properties through several numerical examples and provide a theoretical discussion of conditions for asymptotically superior relative efficiency of the proposed raking estimator.

Raking and regression calibration: Methods to address bias from correlated covariate and time-to-event error

Medical studies that depend on electronic health records (EHR) data are often subject to measurement error, as the data are not collected to support research questions under study. These data errors, if not accounted for in study analyses, can obscure or cause spurious associations between patient exposures and disease risk. Methodology to address covariate measurement error has been well developed; however, time-to-event error has also been shown to cause significant bias, but methods to address it are relatively underdeveloped. More generally, it is possible to observe errors in both the covariate and the time-to-event outcome that are correlated. We propose regression calibration (RC) estimators to simultaneously address correlated error in the covariates and the censored event time. Although RC can perform well in many settings with covariate measurement error, it is biased for nonlinear regression models, such as the Cox model. Thus, we additionally propose raking estimators which are consistent estimators of the parameter defined by the population estimating equation. Raking can improve upon RC in certain settings with failure-time data, require no explicit modeling of the error structure, and can be utilized under outcome-dependent sampling designs. We discuss features of the underlying estimation problem that affect the degree of improvement the raking estimator has over the RC approach. Detailed simulation studies are presented to examine the performance of the proposed estimators under varying levels of signal, error, and censoring. The methodology is illustrated on observational EHR data on HIV outcomes from the Vanderbilt Comprehensive Care Clinic.

Characterization and Correction of Bias Due to Nonparticipation and the Degree of Loyalty in Large-Scale Finnish Loyalty Card Data on Grocery Purchases: Cohort Study

Background

To date, the evaluation of diet has mostly been based on questionnaires and diaries that have their limitations in terms of being time and resource intensive, and a tendency toward social desirability. Loyalty card data obtained in retailing provides timely and objective information on diet-related behaviors. In Finland, the market is highly concentrated, which provides a unique opportunity to investigate diet through grocery purchases.

Objective

The aims of this study were as follows: (1) to investigate and quantify the selection bias in large-scale (n=47,066) loyalty card (LoCard) data and correct the bias by developing weighting schemes and (2) to investigate how the degree of loyalty relates to food purchases.

Methods

Members of a loyalty card program from a large retailer in Finland were contacted via email and invited to take part in the study, which involved consenting to the release of their grocery purchase data for research purposes. Participants’ sociodemographic background was obtained through a web-based questionnaire and was compared to that of the general Finnish adult population obtained via Statistics Finland. To match the distributions of sociodemographic variables, poststratification weights were constructed by using the raking method. The degree of loyalty was self-estimated on a 5-point rating scale.

Results

On comparing our study sample with the general Finnish adult population, in our sample, there were more women (65.25%, 30,696/47,045 vs 51.12%, 2,273,139/4,446,869), individuals with higher education (56.91%, 20,684/36,348 vs 32.21%, 1,432,276/4,446,869), and employed individuals (60.53%, 22,086/36,487 vs 52.35%, 2,327,730/4,446,869). Additionally, in our sample, there was underrepresentation of individuals aged under 30 years (14.44%, 6,791/47,045 vs 18.04%, 802,295/4,446,869) and over 70 years (7.94%, 3,735/47,045 vs 18.20%, 809,317/4,446,869), as well as retired individuals (23.51%, 8,578/36,487 vs 31.82%, 1,414,785/4,446,869). Food purchases differed by the degree of loyalty, with higher shares of vegetable, red meat & processed meat, and fat spread purchases in the higher loyalty groups.

Conclusions

Individuals who consented to the use of their loyalty card data for research purposes tended to diverge from the general Finnish adult population. However, the high volume of data enabled the inclusion of sociodemographically diverse subgroups and successful correction of the differences found in the distributions of sociodemographic variables. In addition, it seems that food purchases differ according to the degree of loyalty, which should be taken into account when researching loyalty card data. Despite the limitations, loyalty card data provide a cost-effective approach to reach large groups of people, including hard-to-reach population subgroups.

Creating National Weights for a Patient-level Longitudinal Database

Objective: To create a nationally-representative estimate from longitudinal data by controlling for sociodemographic factors and health status. Method: The Agency for Healthcare Research and Quality's (AHRQ) Medicare Expenditures Panel Survey (MEPS) was used as the basis for adjustment methodology. MEPS is a data source representing health insurance coverage cost and utilization, and comprises several large-scale surveys of families, individuals, employers, and health care providers. Using these data, we created subset populations. We then used multivariate logistic regression to construct demographics and case-mix-based weights, which were applied to create a population sample that is similar to the national population. The weight was derived using the inverse probability of the weighting approach, as well as a raking mechanism. We compared the results with the projected number of persons in the US population in the same categories to examine the validity of the weights. Results: The following variables were used in the logistic regression: Age group, gender, race, location, income level and health status (Charlson Comorbidity Index scores and chronic condition diagnosis). Relative to MEPS data, patients included in the private insurance data were more likely to be male, older, to have a chronic condition, and to be white (p=0.0000). Adjusted weighted values for patients in the commercial group ranged from 15.47 to 36.36 (median: 16.91). Commercial insurance and MEPS data populations were similar in terms of their socioeconomic and clinical categories. As an outcomes measure, the predicted annual number of patients with prescription claims from private insurance data was 6 963 034. The annual number of statin users were predicted as 6 709 438 using weighted MEPS data. Conclusion: National projections of large-scale patient longitudinal databases require adjustment utilizing demographic factors and case-mix differences related to health status.

Effect of Asthma Call-back Survey methodology changes on work-related asthma estimates, 19 states, 2007-2012

OBJECTIVE

Asthma Call-back Survey methodology has been changed recently, as a new sampling design, weights calculation (2011-2012), and revised work-related asthma (WRA) section (2012) were implemented. To assess the effect of these changes on the WRA and possible WRA estimates among ever-employed adults with current asthma, we analyzed 2007-2012 data for 37 505 ever-employed adults (≥18 years) collected from 19 US states (representing an estimated 10 million adults each year).

METHODS

Using data from landline telephone (LLP) households, we calculated estimates applying poststratification weights (2007-2010) and "raking" weights (2011-2012). Also, using data from LLP/cellular telephone (CP) households combined, we calculated estimates applying "raking" weights (2012).

RESULTS

Based on LLP household data, the WRA estimates ranged from 7.8% to 9.7% during 2007-2010, was 9.1% in 2011 and 15.4% in 2012. Possible WRA estimates ranged from 35.1% to 38.1% during 2007-2010, was 38.1% in 2011 and 39.8% in 2012. Using the 2012 LLP/CP household data, the WRA and possible WRA estimates were 15.4% and 38.9%, respectively.

CONCLUSIONS

Implementation of "raking" weights did not substantially change the WRA or possible WRA estimates among ever-employed adults with current asthma. The WRA and possible WRA estimates based on LLP and LLP/CP samples in 2012 were comparable, as CP users are younger and less likely to have WRA. The substantial upward shift in the 2012 WRA estimates likely was associated with the revision to the WRA section.