Current use of effect size or confidence interval analyses in clinical and biomedical research

Reporting of confidence intervals, achievement of intended sample size, and adjustment for multiple primary outcomes in randomised trials of physical therapy interventions: an analysis of 100 representatively sampled trials

BACKGROUND

The physical therapy profession has made efforts to increase the use of confidence intervals due to the valuable information they provide for clinical decision-making. Confidence intervals indicate the precision of the results and describe the strength and direction of a treatment effect measure.

OBJECTIVES

To determine the prevalence of reporting of confidence intervals, achievement of intended sample size, and adjustment for multiple primary outcomes in randomised trials of physical therapy interventions.

METHODS

We randomly selected 100 trials published in 2021 and indexed on the Physiotherapy Evidence Database. Two independent reviewers extracted the number of participants, any sample size calculation, and any adjustments for multiple primary outcomes. We extracted whether at least one between-group comparison was reported with a 95 % confidence interval and whether any confidence intervals were interpreted.

RESULTS

The prevalence of use of confidence intervals was 47 % (95 % CI 38, 57). Only 6 % of trials (95 % CI: 3, 12) both reported and interpreted a confidence interval. Among the 100 trials, 59 (95 % CI: 49, 68) calculated and achieved the required sample size. Among the 100 trials, 19 % (95 % CI: 13, 28) had a problem with unadjusted multiplicity on the primary outcomes.

CONCLUSIONS

Around half of trials of physical therapy interventions published in 2021 reported confidence intervals around between-group differences. This represents an increase of 5 % from five years earlier. Very few trials interpreted the confidence intervals. Most trials reported a sample size calculation, and among these most achieved that sample size. There is still a need to increase the use of adjustment for multiple comparisons.

The buffet challenge: a behavioral assessment of eating behavior in adolescents with an eating disorder

OBJECTIVE

Eating disorders are characterized by disturbances in nutritional intake and abnormal mealtime behaviors. Laboratory eating paradigms offer a unique opportunity to accurately measure dietary intake and eating behaviors, however, these studies have predominantly occurred in adults. This paper describes the development and preliminary psychometric examination of the Buffet Challenge, a laboratory-based meal task for youths with an eating disorder.

METHOD

We recruited and assessed 56 participants as part of a randomized controlled trial of Family-Based Treatment for adolescents with anorexia nervosa. Adolescents completed the Buffet Challenge at baseline, midway through treatment (~ week 16 of a 6 months course), and end of treatment. Participants and their parents also reported eating disorder symptomatology and treatment related variables of interest were recorded.

RESULTS

All adolescents were willing to complete the Buffet Challenge at all time points, although one refused to give up their cellphone, and there were no significant adverse events recorded. Preliminary results are presented.

CONCLUSIONS

Our initial pilot of this task in adolescents with anorexia nervosa demonstrates its acceptability, although investigation of our hypotheses was hindered by significant missing data due to COVID-related research shutdowns. Future studies should replicate procedures in a larger sample to ensure analyses are adequately powered.

Effect sizes and effect size benchmarks in family violence research

Scholarly journals increasingly request that authors include effect size (ES) estimates when reporting statistical results. However, there is little guidance on how authors should interpret ESs. Consequently, some authors do not provide ES interpretations, or, when interpretations are provided, they often fail to use appropriate reference groups, using instead the ES benchmarks suggested by Cohen (1988). After discussing the most commonly used ES estimates, we describe the method used by Cohen (1962) to develop ES benchmarks (i.e., small, medium, and large) for use in power analyses and describe the limitations associated with using these benchmarks. Next, we establish general benchmarks for family violence (FV) research. That is, we followed Cohen's approach to establishing his original ES benchmarks using family violence research published in 2021 in Child Abuse & Neglect, which produced a medium ES (d = 0.354) that was smaller than Cohen's recommended medium ES (d = 0.500). Then, we examined the ESs in different subspecialty areas of FV research to provide benchmarks for contextualized FV ESs and to provide information that can be used to conduct power analyses when planning future FV research. Finally, some of the challenges to developing ES benchmarks in any scholarly discipline are discussed. For professionals who are not well informed about ESs, the present review is designed to increase their understanding of ESs and what ES benchmarks tell them (and do not tell them) with respect to understanding the meaningfulness of FV research findings.

Accurate Confidence and Bayesian Interval Estimation for Non-centrality Parameters and Effect Size Indices

Reporting effect size index estimates with their confidence intervals (CIs) can be an excellent way to simultaneously communicate the strength and precision of the observed evidence. We recently proposed a robust effect size index (RESI) that is advantageous over common indices because it's widely applicable to different types of data. Here, we use statistical theory and simulations to develop and evaluate RESI estimators and confidence/credible intervals that rely on different covariance estimators. Our results show (1) counter to intuition, the randomness of covariates reduces coverage for Chi-squared and F CIs; (2) when the variance of the estimators is estimated, the non-central Chi-squared and F CIs using the parametric and robust RESI estimators fail to cover the true effect size at the nominal level. Using the robust estimator along with the proposed nonparametric bootstrap or Bayesian (credible) intervals provides valid inference for the RESI, even when model assumptions may be violated. This work forms a unified effect size reporting procedure, such that effect sizes with confidence/credible intervals can be easily reported in an analysis of variance (ANOVA) table format.