A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects

Digital mental health interventions for the treatment of depression: A multiverse meta-analysis

BACKGROUND

The varying sizes of effects in published meta-analyses on digital interventions for depression prompt questions about their efficacy.

METHODS

A systematic search in Embase, PsycINFO, and PubMed identified 125 randomised controlled trials up to February 2023, comparing digital interventions for depression against inactive controls. The stability of results was evaluated with a multiverse meta-analysis, thousands of meta-analyses were conducted based on different combinations of analytical choices, like target populations, intervention characteristics, and study designs.

RESULTS

A total of 3638 meta-analyses were performed based on 125 randomised controlled trials and 263 effect sizes, with a total of 32,733 participants. The average effect size was Hedges' g = 0.43, remaining positive at both the 10th (g = 0.16) and 90th percentiles (g = 0.74). Most meta-analyses indicated a statistically significant benefit of digital interventions. Larger effects were observed in meta-analyses focusing on adults, low- and middle-income countries, guided interventions, comparing interventions with waitlist controls, and patients with major depressive or unipolar mood disorders. Smaller effects appeared when adjusting for publication bias and in assessments after 24 weeks.

LIMITATIONS

While multiverse meta-analysis aims to exhaustively investigate various analytical decisions, some subjectivity remains due to the necessity of making choices that affect the methodology. Additionally, the quality of the included primary studies was low.

CONCLUSIONS

The analytical decisions made during performing pairwise meta-analyses result in vibrations from small to medium effect sizes. Our study provides robust evidence for the effectiveness of digital interventions for depression while highlighting important factors associated with treatment outcomes.

Theories and hypotheses: The forgotten plane of the multiverse

Multiverse analyses-the systematic examination of the effects of decisions that researchers can take over the course of a research project-became more common in recent psychophysiological research. However, multiverse analyses in psychophysiology almost exclusively focus on methodological and statistical decisions that can have a considerable impact on the findings. The role of the conceptual multiverse regarding theory-related research decisions is largely ignored. We argue that the choice of a theory that guides hypotheses, study design, measurement methods, and statistical analyses is the first plane of the psychophysiological multiverse. Depending on the chosen theoretical framework, researchers will choose different methods, and statistical analyses will emphasize specific aspects. We illustrate this process with a research example studying the effects of task difficulty manipulations on cardiovascular effects reflecting effort. We argue in favor of an approach that explicitly acknowledges the various theoretical accounts that can constitute the background of a study and demonstrate how a comparative analytical approach can provide a comprehensive multiverse without increasing type I error due to mere exploration.

Vibration of effects resulting from treatment selection in mixed-treatment comparisons: a multiverse analysis on network meta-analyses of antidepressants in major depressive disorder

OBJECTIVE

It is frequent to find overlapping network meta-analyses (NMAs) on the same topic with differences in terms of both treatments included and effect estimates. We aimed to evaluate the impact on effect estimates of selecting different treatment combinations (ie, network geometries) for inclusion in NMAs.

DESIGN

Multiverse analysis, covering all possible NMAs on different combinations of treatments.

SETTING

Data from a previously published NMA exploring the comparative effectiveness of 22 treatments (21 antidepressants and a placebo) for the treatment of acute major depressive disorder.

PARTICIPANTS

Cipriani et al explored a dataset of 116 477 patients included in 522 randomised controlled trials.

MAIN OUTCOME MEASURES

For each possible treatment selection, we performed an NMA to estimate comparative effectiveness on treatment response and treatment discontinuation for the treatments included (231 between-treatment comparisons). The distribution of effect estimates of between-treatment comparisons across NMAs was computed, and the direction, magnitude and statistical significance of the 1st and 99th percentiles were compared.

RESULTS

4 116 254 different NMAs concerned treatment response. Among possible network geometries, 172/231 (74%) pairwise comparisons exhibited opposite effects between the 1st and 99th percentiles, 57/231 (25%) comparisons exhibited statistically significant results in opposite directions, 118 of 231 (51%) comparisons derived results that were both significant and non-significant at 5% risk and 56/231 (24%) treatment pairs obtained consistent results with only significant differences (or only non-significant differences) at 5% risk. Comparisons based on indirect evidence only were associated with greater variability in effect estimates. Comparisons with small absolute values observed in the complete NMA more frequently obtained statistically significant results in opposite directions. Similar results were observed for treatment discontinuation.

CONCLUSION

In this multiverse analysis, we observed that the selection of treatments to be included in an NMA could have considerable consequences on treatment effect estimations.

TRIAL REGISTRATION

https://osf.io/mb5dy.

Replicable and robust cellular and biochemical blood marker signatures of depression and depressive symptoms

BACKGROUND

Identification of replicable and robust peripheral blood-based markers associated with depression remains elusive, given that studies frequently identify potential biomarkers that ultimately fail to replicate in other studies, impeding progress in psychiatric research. Peripheral biochemical and cellular markers (PBCs; e.g., albumin) may play an important role in depression.

METHODS

Using a test-replication design including participants from the NHANES community cohort (ntest=17,450, nreplication=17,449), we examined 42 PBCs to identify PBCs that were both replicably and robustly associated with either overall depression severity or individual symptoms of depression across both cohorts across a wide range of possible combinations of analytic decisions (n's = 17,000+).

RESULTS

We found that a small set of PBCs (e.g., bilirubin) were robustly and replicably associated with overall depression severity, with unique signatures of PBCs linked with individual symptoms of depression when stratified by gender. A varying degree of correlation was found between measures of replicability.

CONCLUSIONS

We identified replicable and robust cellular biochemical blood marker signatures associated with both overall depression severity and individual symptoms of depression. Our findings can be used to enhance other researchers' abilities to better understand factors associated with depression and potentially drive the development of effective treatments for depression.

A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism

The number of meta-analyses of aggregate data has dramatically increased due to the facility of obtaining data from publications and the development of free, easy-to-use, and specialised statistical software. Even when meta-analyses include the same studies, their results may vary owing to different methodological choices. Assessment of the replication of meta-analysis provides an example of the variation of effect 'naturally' observed between multiple research projects. Reproducibility of results has mostly been reported using graphical descriptive representations. A quantitative analysis of such results would enable (i) breakdown of the total observed variability with quantification of the variability generated by the replication process and (ii) identification of which variables account for this variability, such as methodological quality or the statistical analysis procedures used. These variables might explain systematic mean differences between results and dispersion of the results. To quantitatively characterise the reproducibility of meta-analysis results, a bivariate linear mixed-effects model was developed to simulate both mean results and their corresponding uncertainty. Results were assigned to several replication groups, those assessing the same studies, outcomes, treatment indication and comparisons classified in the same replication group. A nested random effect structure was used to break down the total variability within each replication group and between these groups to enable calculation of an intragroup correlation coefficient and quantification of reproducibility. Determinants of variability were investigated by modelling both mean and variance parameters using covariates. The proposed model was applied to the example of meta-analyses evaluating direct oral anticoagulants in the acute treatment of venous thromboembolism.

Traditional Chinese medicine for smoking cessation: An umbrella review of systematic reviews and meta-analysis of randomized controlled trials

INTRODUCTION

Traditional Chinese medicine (TCM) may have special advantages in facilitating smoking cessation, but consensus on effectiveness is lacking. We aim to comprehensively review, update, and refine current evidence on TCM effectiveness and safety.

METHODS

Nine databases were searched from their inception up to 28 February 2023. Systematic reviews (SRs) and meta-analysis of TCM for smoking cessation were identified and retrieved. Additional databases and hand searches of RCTs from included SRs were performed for data pooling. Cochrane ROB tools and AMSTAR-2 were used to evaluate the methodological quality of RCTs and SRs, respectively. RCT data are presented as relative risks (RR) or mean differences (MD) with 95% confidence intervals (CI) using RevMan 5.4.

RESULTS

Thirteen SRs involving 265 studies with 33081 participants were included. Among these 265 studies, 157 were duplicates (58.36%) and 52 were non-RCTs (19.62%). Combined with the remaining 56 RCTs identified through hand searches, 88 RCTs involving 12434 participants were finally included for data synthesis. All the SRs focused on acupoint stimulation, and the majority were of low or very low quality. The methodological quality of RCTs was either unclear or high risk. For continuous abstinence rate, TCM external interventions were better than placebo in 6 months to 1 year (RR=1.60; 95% CI: 1.14-2.25; I2=27%; n=5533 participants). Compared with placebo, TCM external application was effective in reducing nicotine withdrawal symptoms, and the effect was gradually stable and obvious in the fourth week (MD= -4.46; 95% CI: -5.43 - -3.49; n=165 participants). Twelve RCTs reported adverse events as outcome indicators for safety evaluation, and no serious adverse events occurred.

CONCLUSIONS

Despite the methodological limitations of the original studies, our review suggests that TCM intervention shows potential effectiveness on the continuous abstinence rate. Extending the intervention time can enhance the effect of TCM on nicotine withdrawal symptoms. Referred to adverse events, more data for safety evaluation are required.

Exploring the multiverse: the impact of researchers' analytic decisions on relationships between depression and inflammatory markers

In recent years, a replication crisis in psychiatry has led to a growing focus on the impact of researchers' analytic decisions on the results from studies. Multiverse analyses involve examining results across a wide array of possible analytic decisions (e.g., log-transforming variables, number of covariates, or treatment of outliers) and identifying if study results are robust to researchers' analytic decisions. Studies have begun to use multiverse analysis for well-studied relationships that have some heterogeneity in results/conclusions across studies.We examine the well-studied relationship between peripheral inflammatory markers (PIMs; e.g., white blood cell count (WBC) and C-reactive protein (CRP)) and depression severity in the large NHANES dataset (n = 25,962). Specification curve analyses tested the impact of 9 common analytic decisions (comprising of 58,000+ possible combinations) on the association of PIMs and depression severity. Relationships of PIMs and total depression severity are robust to analytic decisions (based on tests of inference jointly examining effect sizes and p-values). However, moderate/large differences are noted in effect sizes based on analytic decisions and the majority of analyses do not result in significant findings, with the percentage of analyses with statistically significant results being 46.1% for WBC and 43.8% for CRP. For associations of PIMs with specific symptoms of depression, some associations (e.g., sleep, appetite) in males (but not females) were robust to analytic decisions. We discuss how multiverse analyses can be used to guide research and also the need for authors, reviewers, and editors to incorporate multiverse analyses to enhance replicability of research findings.

A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results

OBJECTIVES

To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures.

STUDY DESIGN AND SETTING

A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates.

RESULTS

We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%-99%; quality of life 58%-76%; adverse events 87%-91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%-78%, quality of life; 10%-24%; adverse events 10%-13%).

CONCLUSION

Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.