A comparison of methods for analysing multiple outcome measures in randomised controlled trials using a simulation study

The impact of measurement on clinical trials: Comparison of preliminary outcomes of a brief mobile intervention for autistic adults using multiple measurement approaches

Initial studies of the emotional safety plan (ESP), a new, brief telehealth and mobile intervention to support autistic adults to cope with periods of distress, have reported feasibility and acceptability (Bal et al., 2023, Autism, 1-13). Herein we report the preliminary clinical outcomes of thirty-six autistic adults who developed a personalized ESP, with a specific interest in comparing "outcomes" demonstrated by different instruments and assessment frequencies in order to inform outcome measurement in future clinical trials. Comparison of pre-intervention baseline to post-monitoring outcome (pre-post) anxiety symptoms (Generalized Anxiety Disorder-7 [GAD-7]) and depressive symptoms (Patient Health Questionnaire-9 [PHQ-9]) revealed medium effect sizes for reduction in symptoms, though, low effect sizes were observed for pre-post Adult Self-report Anxiety and Depressive Problems scales and the Emotion Dysregulation Inventory Reactivity and Dysphoria scales. Weekly assessments showed an initial decrease in GAD-7 anxiety symptoms but no effect on weekly PHQ-9 depressive ratings. Ecological momentary assessment (EMA) data suggested that, when participants reported feeling sad or agitated and used the ESP, reduced negative feelings and increased positive states were reported in subsequent EMA. Perhaps not surprisingly, preliminary outcomes of these feasibility trials differed depending on measure chosen. Implications for the design of clinical trials are discussed.

Gaining more from doing less? The effects of a one-week deload period during supervised resistance training on muscular adaptations

Background

Based on emerging evidence that brief periods of cessation from resistance training (RT) may re-sensitize muscle to anabolic stimuli, we aimed to investigate the effects of a 1-week deload interval at the midpoint of a 9-week RT program on muscular adaptations in resistance-trained individuals.

Methods

Thirty-nine young men (n = 29) and women (n = 10) were randomly assigned to 1 of 2 experimental, parallel groups: An experimental group that abstained from RT for 1 week at the midpoint of a 9-week, high-volume RT program (DELOAD) or a traditional training group that performed the same RT program continuously over the study period (TRAD). The lower body routines were directly supervised by the research staff while upper body training was carried out in an unsupervised fashion. Muscle growth outcomes included assessments of muscle thickness along proximal, mid and distal regions of the middle and lateral quadriceps femoris as well as the mid-region of the triceps surae. Adaptions in lower body isometric and dynamic strength, local muscular endurance of the quadriceps, and lower body muscle power were also assessed.

Results

Results indicated no appreciable differences in increases of lower body muscle size, local endurance, and power between groups. Alternatively, TRAD showed greater improvements in both isometric and dynamic lower body strength compared to DELOAD. Additionally, TRAD showed some slight psychological benefits as assessed by the readiness to train questionnaire over DELOAD.

Conclusion

In conclusion, our findings suggest that a 1-week deload period at the midpoint of a 9-week RT program appears to negatively influence measures of lower body muscle strength but has no effect on lower body hypertrophy, power or local muscular endurance.

Incorporating Data from Multiple Endpoints in the Analysis of Clinical Trials: Example from RSV Vaccines

BACKGROUND

To meet regulatory approval, interventions must demonstrate efficacy against a primary outcome in randomized clinical trials. However, when there are multiple clinically relevant outcomes, selecting a single primary outcome is challenging. Incorporating data from multiple outcomes may increase statistical power in clinical trials. We examined methods for analyzing data on multiple endpoints, inspired by real-world trials of interventions against respiratory syncytial virus (RSV).

METHOD

We developed a novel permutation test representing a weighted average of individual outcome test statistics ( wavP ) to evaluate intervention efficacy in a multiple endpoint analysis. We compared the power and type I error rate of this approach to the Bonferroni correction ( bonfT ) and the minP permutation test. We evaluated the different approaches using simulated data from three hypothetical trials varying the intervention efficacy, correlation, and incidence of the outcomes, and data from a real-world RSV clinical trial.

RESULTS

When the vaccine efficacy against different outcomes was similar, wavP yielded higher power than bonfT and minP ; in some scenarios the improvement in power was substantial. In settings where vaccine efficacy was notably larger against one endpoint compared with the others, all three methods had similar power. We developed an R package, PERmutation basEd ANalysis of mulTiple Endpoints (PERMEATE), to guide the selection of the most appropriate method for analyzing multiple endpoints in clinical trials.

CONCLUSIONS

Analyzing multiple endpoints using a weighted permutation method can increase power, whereas controlling the type I error rate compared with established methods under conditions mirroring real-world RSV clinical trials.

Heart Rate Characteristics Monitoring for Late-Onset Sepsis in Preterm Infants: A Systematic Review

BACKGROUND

Early diagnosis of late-onset sepsis (LOS) and necrotizing enterocolitis (NEC) by monitoring heart rate characteristics (HRC) of preterm infants might reduce the risk of death and morbidities. We aimed to systematically assess the effects of HRC monitoring on death, LOS, and NEC.

METHODS

A systematic search was performed in MEDLINE, Embase, Cochrane Library, and Web of Science.

RESULTS

Fifteen papers were included in this review. Three of these papers reported results from the only identified randomized controlled trial (RCT). This RCT showed that HRC monitoring resulted in a small but significant reduction in mortality (absolute risk reduction 2.1% [95% confidence interval 0.01-4.14]) without any differences in neurodevelopmental impairment. The risk of bias was rated high due to performance and detection bias and failure to correct for multiple testing. Most diagnostic cohort studies showed high discriminating accuracy in predicting LOS but lacked sufficient quality and generalizability. No studies for the detection of NEC were identified.

CONCLUSION

Supported by multiple observational cohort studies, the RCT identified in this systematic review showed that HRC monitoring as an early warning system for LOS might reduce the risk of death in preterm infants. However, methodological weaknesses and limited generalizability do not justify implementation of HRC in clinical care. A large international RCT is warranted.

Long-term impacts of an urban sanitation intervention on enteric pathogens in children in Maputo city, Mozambique: study protocol for a cross-sectional follow-up to the Maputo Sanitation (MapSan) trial 5 years postintervention

INTRODUCTION

We previously assessed the effect of an onsite sanitation intervention in informal neighbourhoods of urban Maputo, Mozambique on enteric pathogen detection in children after 2 years of follow-up (Maputo Sanitation (MapSan) trial, ClinicalTrials.gov: NCT02362932). We found significant reductions in Shigella and Trichuris prevalence but only among children born after the intervention was delivered. In this study, we assess the health impacts of the sanitation intervention after 5 years among children born into study households postintervention.

METHODS AND ANALYSIS

We are conducting a cross-sectional household study of enteric pathogen detection in child stool and the environment at compounds (household clusters sharing sanitation and outdoor living space) that received the pour-flush toilet and septic tank intervention at least 5 years prior or meet the original criteria for trial control sites. We are enrolling at least 400 children (ages 29 days to 60 months) in each treatment arm. Our primary outcome is the prevalence of 22 bacterial, protozoan, and soil transmitted helminth enteric pathogens in child stool using the pooled prevalence ratio across the outcome set to assess the overall intervention effect. Secondary outcomes include the individual pathogen detection prevalence and gene copy density of 27 enteric pathogens (including viruses); mean height-for-age, weight-for-age, and weight-for-height z-scores; prevalence of stunting, underweight, and wasting; and the 7-day period prevalence of caregiver-reported diarrhoea. All analyses are adjusted for prespecified covariates and examined for effect measure modification by age. Environmental samples from study households and the public domain are assessed for pathogens and faecal indicators to explore environmental exposures and monitor disease transmission.

ETHICS AND DISSEMINATION

Study protocols have been reviewed and approved by human subjects review boards at the Ministry of Health, Republic of Mozambique and the University of North Carolina at Chapel Hill. Deidentified study data will be deposited at https://osf.io/e7pvk/.

TRIAL REGISTRATION NUMBER

ISRCTN86084138.

Incorporating data from multiple endpoints in the analysis of clinical trials: example from RSV vaccines

Background

To achieve licensure, interventions typically must demonstrate efficacy against a primary outcome in a randomized clinical trial. However, selecting a single primary outcome a priori is challenging. Incorporating data from multiple and related outcomes might help to increase statistical power in clinical trials. Inspired by real-world clinical trials of interventions against respiratory syncytial virus (RSV), we examined methods for analyzing data on multiple endpoints.

Method

We simulated data from three different populations in which the efficacy of the intervention and the correlation among outcomes varied. We developed a novel permutation-based approach that represents a weighted average of individual outcome test statistics ( varP ) to evaluate intervention efficacy in a multiple endpoint analysis. We compared the power and type I error rate of this approach to two alternative methods: the Bonferroni correction ( bonfT ) and another permutation-based approach that uses the minimum P-value across all test statistics ( minP ).

Results

When the vaccine efficacy against different outcomes was similar, VarP yielded higher power than bonfT and minP; in some scenarios the improvement in power was substantial. In settings where vaccine efficacy was notably larger against one endpoint compared to the others, all three methods had similar power.

Conclusions

Analyzing multiple endpoints using a weighted permutation method can increase power while controlling the type I error rate in settings where outcomes share similar characteristics, like RSV outcomes. We developed an R package, PERMEATE , to guide selection of the most appropriate method for analyzing multiple endpoints in clinical trials.

Is telerehabilitation an effective maintenance strategy for patients with chronic obstructive pulmonary diseases: a systematic review

BACKGROUND

Pulmonary rehabilitation (PR) has proven to improve the physical and psychosocial function in patients with chronic obstructive pulmonary disease (COPD). However, the gains achieved during pulmonary rehabilitation diminish over time without an effective maintenance strategy. With several factors affecting access to pulmonary rehabilitation, calls for innovative models were made, which saw the emergence of studies exploring telerehabilitation (TR) as an alternative to traditional pulmonary rehabilitation models. Although there are current reviews exploring the effectiveness of telerehabilitation as an alternative for conventional PR, no review has considered telerehabilitation effectiveness in the long term. Hence, this review aims at examining the effectiveness of telerehabilitation following to pulmonary rehabilitation in patients with chronic obstructive pulmonary disease.

MAIN BODY

A systematic review of the literature using CINAHL, MEDLINE, SCOPUS, Web of science PEDRO, AMED and EMBASE databases was conducted to assess the effectiveness of telerehabilitation following PR in patients with COPD. Health-related quality of life (HRQoL) and exercise capacity was maintained within 6-12 months of a TR maintenance programme. However, there was no significant increase in HRQoL and exercise capacity between the intervention and control groups in 6-12 months.

CONCLUSIONS

This review suggests that a TR maintenance strategy effectively maintains benefits gained and may improve HRQoL and exercise capacity within 6-12 months for patients with COPD. Nonetheless, it is impossible to extrapolate the findings to the general population due to the paucity of included studies. Further high quality randomised controlled trials examining TR in the long-term is required in the future.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1186/s42269-023-00980-8.

Using multiple outcomes in intervention studies: improving power while controlling type I errors

Background The CONSORT guidelines for clinical trials recommend using a single primary outcome, to guard against excess false positive findings when multiple measures are considered. However, statistical power can be increased while controlling the familywise error rate if multiple outcomes are included. The MEff statistic is well-suited to this purpose, but is not well-known outside genetics. Methods Data were simulated for an intervention study, with a given sample size (N), effect size (E) and correlation matrix for a suite of outcomes ( R). Using the variance of eigenvalues from the correlation matrix, we compute MEff, the effective number of variables that the alpha level should be divided by to control the familywise error rate. Various scenarios are simulated to consider how MEff is affected by the pattern of pairwise correlations within a set of outcomes. The power of the MEff approach is compared to Bonferroni correction, and a principal component analysis (PCA). Results In many situations, power can be increased by inclusion of multiple outcomes. Differences in power between MEff and Bonferroni correction are small if intercorrelations between outcomes are low, but the advantage of MEff is more evident as intercorrelations increase. PCA is superior in cases where the impact on outcomes is fairly uniform, but MEff is applicable when intervention effects are inconsistent across measures. Conclusions The optimal method for correcting for multiple testing depends on the underlying data structure, with PCA being superior if outcomes are all indicators of a common underlying factor. Both Bonferroni correction and MEff can be applied post hoc to evaluate published intervention studies, with MEff being superior when outcomes are moderately or highly correlated. A lookup table is provided to give alpha levels for use with Meff for cases where the correlation between outcome measures can be estimated.

Using multiple outcomes in intervention studies: improving power while controlling type I errors

Background The CONSORT guidelines for clinical trials recommend use of a single primary outcome, to guard against the raised risk of false positive findings when multiple measures are considered. It is, however, possible to include a suite of multiple outcomes in an intervention study, while controlling the familywise error rate, if the criterion for rejecting the null hypothesis specifies that N or more of the outcomes reach an agreed level of statistical significance, where N depends on the total number of outcome measures included in the study, and the correlation between them. Methods Simulations were run, using a conventional null-hypothesis significance testing approach with alpha set at .05, to explore the case when between 2 and 12 outcome measures are included to compare two groups, with average correlation between measures ranging from zero to .8, and true effect size ranging from 0 to .7. In step 1, a table is created giving the minimum N significant outcomes (MinNSig) that is required for a given set of outcome measures to control the familywise error rate at 5%. In step 2, data are simulated using MinNSig values for each set of correlated outcomes and the resulting proportion of significant results is computed for different sample sizes,correlations, and effect sizes. Results The Adjust NVar approach can achieve a more efficient trade-off between power and type I error rate than use of a single outcome when there are three or more moderately intercorrelated outcome variables. Conclusions Where it is feasible to have a suite of moderately correlated outcome measures, then this might be a more efficient approach than reliance on a single primary outcome measure in an intervention study. In effect, it builds in an internal replication to the study. This approach can also be used to evaluate published intervention studies.

A comparison of methods for analysing multiple outcome measures in randomised controlled trials using a simulation study

Multiple primary outcomes are sometimes collected and analysed in randomised controlled trials (RCTs), and are used in favour of a single outcome. By collecting multiple primary outcomes, it is possible to fully evaluate the effect that an intervention has for a given disease process. A simple approach to analysing multiple outcomes is to consider each outcome separately, however, this approach does not account for any pairwise correlations between the outcomes. Any cases with missing values must be ignored, unless an additional imputation step is performed. Alternatively, multivariate methods that explicitly model the pairwise correlations between the outcomes may be more efficient when some of the outcomes have missing values. In this paper, we present an overview of relevant methods that can be used to analyse multiple outcome measures in RCTs, including methods based on multivariate multilevel (MM) models. We perform simulation studies to evaluate the bias in the estimates of the intervention effects and the power of detecting true intervention effects observed when using selected methods. Different simulation scenarios were constructed by varying the number of outcomes, the type of outcomes, the degree of correlations between the outcomes and the proportions and mechanisms of missing data. We compare multivariate methods to univariate methods with and without multiple imputation. When there are strong correlations between the outcome measures (ρ > .4), our simulation studies suggest that there are small power gains when using the MM model when compared to analysing the outcome measures separately. In contrast, when there are weak correlations (ρ < .4), the power is reduced when using univariate methods with multiple imputation when compared to analysing the outcome measures separately.