A comparison of different measures of the proportion of explained variance in multiply imputed data sets

The proportion of explained variance is an important statistic in multiple regression for determining how well the outcome variable is predicted by the predictors. Earlier research on 20 different estimators for the proportion of explained variance, including the exact Olkin-Pratt estimator and the Ezekiel estimator, showed that the exact Olkin-Pratt estimator produced unbiased estimates, and was recommended as a default estimator. In the current study, the same 20 estimators were studied in incomplete data, with missing data being treated using multiple imputation. In earlier research on the proportion of explained variance in multiply imputed data sets, an estimator calledR ̂ SP 2 $$ {\hat{R}}_{\mathrm{SP}}^2 $$ was shown to be the preferred pooled estimator for regularR 2 $$ {R}^2 $$ . For each of the 20 estimators in the current study, two pooled estimators were proposed: one where the estimator was the average across imputed data sets, and one whereR ̂ SP 2 $$ {\hat{R}}_{\mathrm{SP}}^2 $$ was used as input for the calculation of the specific estimator. Simulations showed that estimates based onR ̂ SP 2 $$ {\hat{R}}_{\mathrm{SP}}^2 $$ performed best regarding bias and accuracy, and that the Ezekiel estimator was generally the least biased. However, none of the estimators were unbiased at all times, including the exact Olkin-Pratt estimator based onR ̂ SP 2 $$ {\hat{R}}_{\mathrm{SP}}^2 $$ .

Cost-effectiveness of three internet-based interventions for eating disorders: A randomized controlled trial

OBJECTIVE

The primary aim was assessing the cost-effectiveness of an internet-based self-help program, expert-patient support, and the combination of both compared to a care-as-usual condition.

METHOD

An economic evaluation from a societal perspective was conducted alongside a randomized controlled trial. Participants aged 16 or older with at least mild eating disorder symptoms were randomly assigned to four conditions: (1) Featback, an online unguided self-help program, (2) chat or e-mail support from a recovered expert patient, (3) Featback with expert-patient support, and (4) care-as-usual. After a baseline assessment and intervention period of 8 weeks, five online assessments were conducted over 12 months of follow-up. The main result constituted cost-utility acceptability curves with quality-of-life adjusted life years (QALYs) and societal costs over the entire study duration.

RESULTS

No significant differences between the conditions were found regarding QALYs, health care costs and societal costs. Nonsignificant differences in QALYs were in favor of the Featback conditions and the lowest societal costs per participant were observed in the Featback only condition (€16,741) while the highest costs were seen in the care-as-usual condition (€28,479). The Featback only condition had the highest probability of being efficient compared to the alternatives for all acceptable willingness-to-pay values.

DISCUSSION

Featback, an internet-based unguided self-help intervention, was likely to be efficient compared to Featback with guidance from an expert patient, guidance alone and a care-as-usual condition. Results suggest that scalable interventions such as Featback may reduce health care costs and help individuals with eating disorders that are currently not reached by other forms of treatment.

PUBLIC SIGNIFICANCE STATEMENT

Internet-based interventions for eating disorders might reach individuals in society who currently do not receive appropriate treatment at low costs. Featback, an online automated self-help program for eating disorders, was found to improve quality of life slightly while reducing costs for society, compared to a do-nothing approach. Consequently, implementing internet-based interventions such as Featback likely benefits both individuals suffering from an eating disorder and society as a whole.

The diagnostic process from primary care to child and adolescent mental healthcare services: the incremental value of information conveyed through referral letters, screening questionnaires and structured multi-informant assessment

BACKGROUND

A variety of information sources are used in the best-evidence diagnostic procedure in child and adolescent mental healthcare, including evaluation by referrers and structured assessment questionnaires for parents. However, the incremental value of these information sources is still poorly examined.

AIMS

To quantify the added and unique predictive value of referral letters, screening, multi-informant assessment and clinicians' remote evaluations in predicting mental health disorders.

METHOD

Routine medical record data on 1259 referred children and adolescents were retrospectively extracted. Their referral letters, responses to the Strengths and Difficulties Questionnaire (SDQ), results on closed-ended questions from the Development and Well-Being Assessment (DAWBA) and its clinician-rated version were linked to classifications made after face-to-face intake in psychiatry. Following multiple imputations of missing data, logistic regression analyses were performed with the above four nodes of assessment as predictors and the five childhood disorders common in mental healthcare (anxiety, depression, autism spectrum disorders, attention-deficit hyperactivity disorder, behavioural disorders) as outcomes. Likelihood ratio tests and diagnostic odds ratios were computed.

RESULTS

Each assessment tool significantly predicted the classified outcome. Successive addition of the assessment instruments improved the prediction models, with the exception of behavioural disorder prediction by the clinician-rated DAWBA. With the exception of the SDQ for depressive and behavioural disorders, all instruments showed unique predictive value.

CONCLUSIONS

Structured acquisition and integrated use of diverse sources of information supports evidence-based diagnosis in clinical practice. The clinical value of structured assessment at the primary-secondary care interface should now be quantified in prospective studies.

Multiple imputation to balance unbalanced designs for two-way analysis of variance

A balanced ANOVA design provides an unambiguous interpretation of the F-tests, and has more power than an unbalanced design. In earlier literature, multiple imputation was proposed to create balance in unbalanced designs, as an alternative to Type-III sum of squares. In the current simulation study we studied four pooled statistics for multiple imputation, namely D₀, D₁, D₂, and D₃ in unbalanced data, and compared them with Type-III sum of squares. Statistics D₁ and D₂ generally performed best regarding Type-I error rates, and had power rates closest to that of Type-III sum of squares. Additionally, for the interaction, D₁ produced power rates higher than Type-III sum of squares. For multiply imputed datasets D₁ and D₂ may be the best methods for pooling the results in multiply imputed datasets, and for unbalanced data, D₁ might be a good alternative to Type-III sum of squares regarding the interaction.

Much ado about nothing: Multiple imputation to balance unbalanced designs for two-way analysis of variance

In earlier literature, multiple imputation was proposed to create balance in unbalanced designs, as an alternative to Type III sum of squares in two-way ANOVA. In the current simulation study we studied four pooled statistics for multiple imputation, namely D₀, D₁, D₂, and D₃ in unbalanced data, and compared these statistics with Type III sum of squares. Statistics D₀ and D₂ generally performed best regarding Type-I error rates, and had power rates closest to that of Type III sum of squares. However, none of the statistics produced power rates higher than Type III sum of squares. The results lead to the conclusion that for multiply imputed datasets D₀ and D₂ may be the best methods for pooling the results of multiparameter estimates in multiply imputed datasets, and that for unbalanced data, Type III sum of square is to be preferred over using multiple imputation in obtaining ANOVA results.

Intergenerational transmission of child maltreatment using a multi-informant multi-generation family design

In the current study a three-generational design was used to investigate intergenerational transmission of child maltreatment (ITCM) using multiple sources of information on child maltreatment: mothers, fathers and children. A total of 395 individuals from 63 families reported on maltreatment. Principal Component Analysis (PCA) was used to combine data from mother, father and child about maltreatment that the child had experienced. This established components reflecting the convergent as well as the unique reports of father, mother and child on the occurrence of maltreatment. Next, we tested ITCM using the multi-informant approach and compared the results to those of two more common approaches: ITCM based on one reporter and ITCM based on different reporters from each generation. Results of our multi-informant approach showed that a component reflecting convergence between mother, father, and child reports explained most of the variance in experienced maltreatment. For abuse, intergenerational transmission was consistently found across approaches. In contrast, intergenerational transmission of neglect was only found using the perspective of a single reporter, indicating that transmission of neglect might be driven by reporter effects. In conclusion, the present results suggest that including multiple informants may be necessary to obtain more valid estimates of ITCM.

Standardized Regression Coefficients and Newly Proposed Estimators for [Formula: see text] in Multiply Imputed Data

Whenever statistical analyses are applied to multiply imputed datasets, specific formulas are needed to combine the results into one overall analysis, also called combination rules. In the context of regression analysis, combination rules for the unstandardized regression coefficients, the t-tests of the regression coefficients, and the F-tests for testing [Formula: see text] for significance have long been established. However, there is still no general agreement on how to combine the point estimators of [Formula: see text] in multiple regression applied to multiply imputed datasets. Additionally, no combination rules for standardized regression coefficients and their confidence intervals seem to have been developed at all. In the current article, two sets of combination rules for the standardized regression coefficients and their confidence intervals are proposed, and their statistical properties are discussed. Additionally, two improved point estimators of [Formula: see text] in multiply imputed data are proposed, which in their computation use the pooled standardized regression coefficients. Simulations show that the proposed pooled standardized coefficients produce only small bias and that their 95% confidence intervals produce coverage close to the theoretical 95%. Furthermore, the simulations show that the newly proposed pooled estimates for [Formula: see text] are less biased than two earlier proposed pooled estimates.

Significance Tests and Estimates for R2 for Multiple Regression in Multiply Imputed Datasets: A Cautionary Note on Earlier Findings, and Alternative Solutions

Whenever multiple regression is applied to a multiply imputed data set, several methods for combining significance tests for R2 and the change in R2 across imputed data sets may be used: the combination rules by Rubin, the Fisher z-test for R2 by Harel, and F-tests for the change in R2 by Chaurasia and Harel. For pooling R2 itself, Harel proposed a method based on a Fisher z transformation. In the current article, it is argued that the pooled R2 based on the Fisher z transformation, the Fisher z-test for R2 , and the F-test for the change in R2 have some theoretical flaws. An argument is made for using Rubin's method for pooling significance tests for R2 instead, and alternative procedures for pooling R2 are proposed: simple averaging and a pooled R2 constructed from the pooled significance test by Rubin. Simulations show that the Fisher z-test and Chaurasia and Harel's F-tests generally give inflated type-I error rates, whereas the type-I error rates of Rubin's method are correct. Of the methods for pooling the point estimates of R2 no method clearly performs best, but it is argued that the average of R2 's across imputed data set is preferred.

Rebutting Existing Misconceptions About Multiple Imputation as a Method for Handling Missing Data

Missing data is a problem that occurs frequently in many scientific areas. The most sophisticated method for dealing with this problem is multiple imputation. Contrary to other methods, like listwise deletion, this method does not throw away information, and partly repairs the problem of systematic dropout. Although from a theoretical point of view multiple imputation is considered to be the optimal method, many applied researchers are reluctant to use it because of persistent misconceptions about this method. Instead of providing an(other) overview of missing data methods, or extensively explaining how multiple imputation works, this article aims specifically at rebutting these misconceptions, and provides applied researchers with practical arguments supporting them in the use of multiple imputation.

One Drop | Mobile on iPhone and Apple Watch: An Evaluation of HbA1c Improvement Associated With Tracking Self-Care

Background

The One Drop | Mobile app supports manual and passive (via HealthKit and One Drop’s glucose meter) tracking of self-care and glycated hemoglobin A_1c (HbA_1c).

Objective

We assessed the HbA_1c change of a sample of people with type 1 diabetes (T1D) or type 2 diabetes (T2D) using the One Drop | Mobile app on iPhone and Apple Watch, and tested relationships between self-care tracking with the app and HbA_1c change.

Methods

In June 2017, we identified people with diabetes using the One Drop | Mobile app on iPhone and Apple Watch who entered two HbA_1c measurements in the app 60 to 365 days apart. We assessed the relationship between using the app and HbA_1c change.

Results

Users had T1D (n=65) or T2D (n=191), were 22.7% (58/219) female, with diabetes for a mean 8.34 (SD 8.79) years, and tracked a mean 2176.35 (SD 3430.23) self-care activities between HbA_1c entries. There was a significant 1.36% or 14.9 mmol/mol HbA_1c reduction (F=62.60, P<.001) from the first (8.72%, 71.8 mmol/mol) to second HbA_1c (7.36%, 56.9 mmol/mol) measurement. Tracking carbohydrates was independently associated with greater HbA_1c improvement (all P<.01).

Conclusions

Using One Drop | Mobile on iPhone and Apple Watch may favorably impact glycemic control.

One Drop | Mobile: An Evaluation of Hemoglobin A1c Improvement Linked to App Engagement

Background

Three recent reviews evaluated 19 studies testing the hemoglobin A1c (HbA1c) benefit of 16 diabetes apps, including 5 publicly available apps. Most studies relied on small samples and did not link app engagement with outcomes.

Objective

This study assessed both HbA1c change in a large sample of people using the One Drop | Mobile app and associations between app engagement and changes in HbA1c.

Methods

The One Drop | Mobile app for iOS and Android is designed to manually and passively (via Apple HealthKit, Google Fit, and the One Drop | Chrome blood glucose meter) store, track, and share data. Users can schedule medication reminders, view statistics, set goals, track health outcomes, and get data-driven insights. In June 2017, we queried data on people with diabetes using the app who had entered at least 2 HbA1c values in the app >60 and ≤365 days apart. Multiple imputation corrected for missing data. Unadjusted and adjusted mixed effects repeated measures models tested mean HbA1c change by time, diabetes type, and their interaction. Multiple regression models assessed relationships between using the app to track food, activity, blood glucose, and medications and HbA1c change.

Results

The sample (N=1288) included people with type 1 diabetes (T1D) (n=367) or type 2 diabetes (T2D) (n=921) who were 35% female, diagnosed with diabetes for a mean 9.4 (SD 9.9) years, and tracked an average 1646.1 (SD 3621.9) self-care activities in One Drop | Mobile between their first (mean 8.14% [SD 2.06%]) and second HbA1c entry (mean 6.98% [SD 1.1%]). HbA1c values were significantly associated with user-entered average blood glucose 90 days before the second HbA1c entry (rho=.73 to .75, P<.001). HbA1c decreased by an absolute 1.07% (unadjusted and adjusted F=292.03, P<.001) from first to second HbA1c entry. There was a significant interaction between diabetes type and HbA1c. Both groups significantly improved, but users with T2D had a greater HbA1c decrease over time than users with T1D (F=10.54, P<.001). For users with T2D (n=921), HbA1c decreased by an absolute 1.27% (F=364.50, P<.001) from first to second HbA1c entry. Finally, using One Drop | Mobile to record food was associated with greater HbA1c reductions even after adjusting for covariates and after also adjusting for insulin use for users with T2D (all P<.05).

Conclusions

People with T1D and T2D reported a 1.07% to 1.27% absolute reduction in HbA1c during a median 4 months of using the One Drop | Mobile app. Using the app to track self-care was associated with improved HbA1c. More research is needed on the health benefits of publicly available diabetes apps, particularly studies associating app engagement with short- and long-term effects.

Web-Based Fully Automated Self-Help With Different Levels of Therapist Support for Individuals With Eating Disorder Symptoms: A Randomized Controlled Trial

Background

Despite the disabling nature of eating disorders (EDs), many individuals with ED symptoms do not receive appropriate mental health care. Internet-based interventions have potential to reduce the unmet needs by providing easily accessible health care services.

Objective

This study aimed to investigate the effectiveness of an Internet-based intervention for individuals with ED symptoms, called “Featback.” In addition, the added value of different intensities of therapist support was investigated.

Methods

Participants (N=354) were aged 16 years or older with self-reported ED symptoms, including symptoms of anorexia nervosa, bulimia nervosa, and binge eating disorder. Participants were recruited via the website of Featback and the website of a Dutch pro-recovery–focused e-community for young women with ED problems. Participants were randomized to: (1) Featback, consisting of psychoeducation and a fully automated self-monitoring and feedback system, (2) Featback supplemented with low-intensity (weekly) digital therapist support, (3) Featback supplemented with high-intensity (3 times a week) digital therapist support, and (4) a waiting list control condition. Internet-administered self-report questionnaires were completed at baseline, post-intervention (ie, 8 weeks after baseline), and at 3- and 6-month follow-up. The primary outcome measure was ED psychopathology. Secondary outcome measures were symptoms of depression and anxiety, perseverative thinking, and ED-related quality of life. Statistical analyses were conducted according to an intent-to-treat approach using linear mixed models.

Results

The 3 Featback conditions were superior to a waiting list in reducing bulimic psychopathology (d=−0.16, 95% confidence interval (CI)=−0.31 to −0.01), symptoms of depression and anxiety (d=−0.28, 95% CI=−0.45 to −0.11), and perseverative thinking (d=−0.28, 95% CI=−0.45 to −0.11). No added value of therapist support was found in terms of symptom reduction although participants who received therapist support were significantly more satisfied with the intervention than those who did not receive supplemental therapist support. No significant differences between the Featback conditions supplemented with low- and high-intensity therapist support were found regarding the effectiveness and satisfaction with the intervention.

Conclusions

The fully automated Internet-based self-monitoring and feedback intervention Featback was effective in reducing ED and comorbid psychopathology. Supplemental therapist support enhanced satisfaction with the intervention but did not increase its effectiveness. Automated interventions such as Featback can provide widely disseminable and easily accessible care. Such interventions could be incorporated within a stepped-care approach in the treatment of EDs and help to bridge the gap between mental disorders and mental health care services.

Trial Registration

Netherlands Trial Registry: NTR3646; http://www.trialregister.nl/trialreg/admin/ rctview.asp?TC=3646 (Archived by WebCite at http://www.webcitation.org/6fgHTGKHE)

Do internationally adopted children in the Netherlands use more medication than their non-adopted peers?

Empirical evidence has shown that international adoptees present physical growth delays, precocious puberty, behavioral problems, and mental health referrals more often than non-adoptees. We hypothesized that the higher prevalence of (mental) health problems in adoptees is accompanied by elevated consumption of prescription drugs, including antidepressants, attention deficit hyperactivity disorder (ADHD) medication, and medication for growth inhibition/stimulation. In an archival, population-based Dutch cohort study, data on medication use were available from the Health Care Insurance Board by Statistics Netherlands from 2006 to 2011. The Dutch population born between 1994 and 2005 and alive during the period of measurement was included (2,360,450 including 10,602 international adoptees, of which 4447 from China). Their mean age was 6.5 years at start (range 1-12 years) and 11.5 years at the end of the measurement period (range 6-17 years). Chinese female adoptees used less medication for precocious puberty (as treatment for precocious puberty; odds ratio (OR) = 0.57, effect size Cohen's d = -0.31) and contraception (OR = 0.65, d = -0.24) than non-adoptees. For both males and females, non-Chinese adoptees used more medication for ADHD than non-adoptees (males: OR = 1.22, females: OR = 1.32), but the effect was small (males: d = 0.11, females: d = 0.15).

CONCLUSIONS

Adoptees in the Netherlands generally do not use more medication than their non-adopted peers.

WHAT IS KNOWN

• Meta-analytical evidence shows that international adoptees present physical growth delays and mental health referrals more often than non-adopted controls. • With the exception of one Swedish study on ADHD medication, there is no other systematic research on medication use of international adoptees. What is New: • All differences in medication use between international adoptees in the Netherlands and non-adopted controls were below the threshold of a small effect with the exception of medication for precocious puberty, but this effect was in the opposite direction with female adoptees using less medication for precocious puberty than non-adoptees. • International adoptees in the Netherlands do not use more medication despite experiences of preadoption adversity and higher rates of mental health referrals during childhood and adolescence.

Analysis of Variance of Multiply Imputed Data

As a procedure for handling missing data, Multiple imputation consists of estimating the missing data multiple times to create several complete versions of an incomplete data set. All these data sets are analyzed by the same statistical procedure, and the results are pooled for interpretation. So far, no explicit rules for pooling F-tests of (repeated-measures) analysis of variance have been defined. In this paper we outline the appropriate procedure for the results of analysis of variance for multiply imputed data sets. It involves both reformulation of the ANOVA model as a regression model using effect coding of the predictors and applying already existing combination rules for regression models. The proposed procedure is illustrated using three example data sets. The pooled results of these three examples provide plausible F- and p-values.

Attachment disorganization moderates the effect of maternal postnatal depressive symptoms on infant autonomic functioning

We examined associations of disorganized attachment and maternal depressive symptoms with infant autonomic functioning in 450 infant-mother dyads enrolled in the Generation R study. Maternal depressive symptoms were measured 2 months postpartum with the Brief Symptom Inventory. At 14 months, we assessed infant attachment with a slightly shortened Strange Situation and measured infant resting heart rate. Respiratory sinus arrhythmia (RSA) was calculated using spectral analysis. Higher levels of maternal postnatal depressive symptoms predicted lower resting RSA in disorganized infants (B = -0.31, SE = 0.15, p = .04, R(2)  = .05) but not in nondisorganized infants (B = 0.05, SE = 0.06, p = .36). This effect was buffered in disorganized infants with a secondary secure attachment classification. Disorganized infants were more vulnerable to the effect of maternal postnatal depressive symptoms on the physiological stress systems.

Constructing bootstrap confidence intervals for principal component loadings in the presence of missing data: a multiple-imputation approach

Earlier research has shown that bootstrap confidence intervals from principal component loadings give a good coverage of the population loadings. However, this only applies to complete data. When data are incomplete, missing data have to be handled before analysing the data. Multiple imputation may be used for this purpose. The question is how bootstrap confidence intervals for principal component loadings should be corrected for multiply imputed data. In this paper, several solutions are proposed. Simulations show that the proposed corrections for multiply imputed data give a good coverage of the population loadings in various situations.

Incidence of Missing Item Scores in Personality Measurement, and Simple Item-Score Imputation

The focus of this study was the incidence of different kinds of missing-data problems in personality research and the handling of these problems. Missing-data problems were reported in approximately half of more than 800 articles published in three leading personality journals. In these articles, unit nonresponse, attrition, and planned missingness were distinguished but missing item scores in trait measurement were reported most frequently. Listwise deletion was the most frequently used method for handling all missing-data problems. Listwise deletion is known to reduce the accuracy of parameter estimates and the power of statistical tests and often to produce biased statistical analysis results. This study proposes a simple alternative method for handling missing item scores, known as two-way imputation, which leaves the sample size intact and has been shown to produce almost unbiased results based on multi-item questionnaire data.

Looking beyond posttraumatic stress disorder in children: posttraumatic stress reactions, posttraumatic growth, and quality of life in a general population sample

OBJECTIVE

In order to broaden the view beyond posttraumatic stress disorder (PTSD) in children, we examined to what extent posttraumatic stress reactions, posttraumatic growth, and quality of life were related to each other and to traumatic exposure in the general population.

METHOD

1770 children of 36 randomly selected primary schools (mean age = 10.24 years, 50% boys) reported in October/November 2006 on their worst experience (traumatic exposure was considered present when the described event fulfilled the A1 criterion for PTSD of the DSM-IV-TR) and filled out the Children's Responses to Trauma Inventory, the Posttraumatic Growth Inventory for Children, and the KIDSCREEN-27. Correlational and hierarchical linear regression analyses were carried out in a multiple imputation format.

RESULTS

Posttraumatic stress reactions were strongly related to posttraumatic growth (r = 0.41, p < .01) and quality of life (r = -0.47, p < .01). The latter 2 variables were weakly related; positively when controlling for posttraumatic stress reactions (r = 0.09, p < .01), negatively when not (r = -0.12, p < .01). Children who were exposed to trauma reported more posttraumatic stress reactions (β = .12, p < .01), more posttraumatic growth (β = .09, p < .01), and less quality of life (β = -.08, p < .01) than nonexposed children (effect sizes were small).

CONCLUSIONS

Negative and positive psychological sequelae of trauma can coexist in children, and extend to broader areas of life than specific symptoms only. Clinicians should look further than PTSD alone and pay attention to the broad range of posttraumatic stress reactions that children show, their experience of posttraumatic growth, and their quality of life.

Multiple imputation for item scores when test data are factorially complex

Multiple imputation under a two-way model with error is a simple and effective method that has been used to handle missing item scores in unidimensional test and questionnaire data. Extensions of this method to multidimensional data are proposed. A simulation study is used to investigate whether these extensions produce biased estimates of important statistics in multidimensional data, and to compare them with lower benchmark listwise deletion, two-way with error and multivariate normal imputation. The new methods produce smaller bias in several psychometrically interesting statistics than the existing methods of two-way with error and multivariate normal imputation. One of these new methods clearly is preferable for handling missing item scores in multidimensional test data.

Multiple Imputation of Item Scores in Test and Questionnaire Data, and Influence on Psychometric Results

The performance of five simple multiple imputation methods for dealing with missing data were compared. In addition, random imputation and multivariate normal imputation were used as lower and upper benchmark, respectively. Test data were simulated and item scores were deleted such that they were either missing completely at random, missing at random, or not missing at random. Cronbach's alpha, Loevinger's scalability coefficient H, and the item cluster solution from Mokken scale analysis of the complete data were compared with the corresponding results based on the data including imputed scores. The multiple-imputation methods, two-way with normally distributed errors, corrected item-mean substitution with normally distributed errors, and response function, produced discrepancies in Cronbach's coefficient alpha, Loevinger's coefficient H, and the cluster solution from Mokken scale analysis, that were smaller than the discrepancies in upper benchmark multivariate normal imputation.