Using multiple outcomes in intervention studies for improved trade-off between power and type I errors: the Adjust NVar approach

Processing Speed Deficit and Its Relationship with Math Fluency in Children with Attention-Deficit/Hyperactivity Disorder

OBJECTIVES

(1) To determine the processing speed (PS) deficit in children with ADHD; and (2) To investigate if PS deficit was the primary cause of daily dysfunction in ADHD by testing the direct and indirect effects via working memory (WM) of PS on math fluency (MF).

METHOD

Seventy-eight children (52 children with ADHD and 26 controls) were tested on their motor, perceptual, cognitive, and verbal PS, WM, and MF.

RESULTS

Children with ADHD performed worse than controls on all PS, suggesting a general PS deficit. Moreover, cognitive PS was a significant predictor for MF. Mediation analysis showed that cognitive PS had direct and indirect effects via WM on MF, suggesting PS deficit might be the primary cause of MF difficulties in ADHD.

CONCLUSION

Findings of this study suggested a general PS deficit in ADHD. Due to the importance of PS in MF, interventions for MF underachievers should include assessment and training of PS.

Using multiple outcomes in intervention studies: improving power while controlling type I errors

Background The CONSORT guidelines for clinical trials recommend using a single primary outcome, to guard against excess false positive findings when multiple measures are considered. However, statistical power can be increased while controlling the familywise error rate if multiple outcomes are included. The MEff statistic is well-suited to this purpose, but is not well-known outside genetics. Methods Data were simulated for an intervention study, with a given sample size (N), effect size (E) and correlation matrix for a suite of outcomes ( R). Using the variance of eigenvalues from the correlation matrix, we compute MEff, the effective number of variables that the alpha level should be divided by to control the familywise error rate. Various scenarios are simulated to consider how MEff is affected by the pattern of pairwise correlations within a set of outcomes. The power of the MEff approach is compared to Bonferroni correction, and a principal component analysis (PCA). Results In many situations, power can be increased by inclusion of multiple outcomes. Differences in power between MEff and Bonferroni correction are small if intercorrelations between outcomes are low, but the advantage of MEff is more evident as intercorrelations increase. PCA is superior in cases where the impact on outcomes is fairly uniform, but MEff is applicable when intervention effects are inconsistent across measures. Conclusions The optimal method for correcting for multiple testing depends on the underlying data structure, with PCA being superior if outcomes are all indicators of a common underlying factor. Both Bonferroni correction and MEff can be applied post hoc to evaluate published intervention studies, with MEff being superior when outcomes are moderately or highly correlated. A lookup table is provided to give alpha levels for use with Meff for cases where the correlation between outcome measures can be estimated.

Using multiple outcomes in intervention studies: improving power while controlling type I errors

Background The CONSORT guidelines for clinical trials recommend use of a single primary outcome, to guard against the raised risk of false positive findings when multiple measures are considered. It is, however, possible to include a suite of multiple outcomes in an intervention study, while controlling the familywise error rate, if the criterion for rejecting the null hypothesis specifies that N or more of the outcomes reach an agreed level of statistical significance, where N depends on the total number of outcome measures included in the study, and the correlation between them. Methods Simulations were run, using a conventional null-hypothesis significance testing approach with alpha set at .05, to explore the case when between 2 and 12 outcome measures are included to compare two groups, with average correlation between measures ranging from zero to .8, and true effect size ranging from 0 to .7. In step 1, a table is created giving the minimum N significant outcomes (MinNSig) that is required for a given set of outcome measures to control the familywise error rate at 5%. In step 2, data are simulated using MinNSig values for each set of correlated outcomes and the resulting proportion of significant results is computed for different sample sizes,correlations, and effect sizes. Results The Adjust NVar approach can achieve a more efficient trade-off between power and type I error rate than use of a single outcome when there are three or more moderately intercorrelated outcome variables. Conclusions Where it is feasible to have a suite of moderately correlated outcome measures, then this might be a more efficient approach than reliance on a single primary outcome measure in an intervention study. In effect, it builds in an internal replication to the study. This approach can also be used to evaluate published intervention studies.