Development of a combined database for meta-epidemiological research

Reevaluation of statistically significant meta-analyses in advanced cancer patients using the Hartung-Knapp method and prediction intervals - a methodological study

Using the Hartung-Knapp method and 95% prediction intervals (PIs) in random-effects meta-analyses is recommended by experts but rarely applied. Therefore, we aimed to reevaluate statistically significant meta-analyses using the Hartung-Knapp method and 95% PIs. In this methodological study three databases were searched from January 2010 to July 2019. We included systematic reviews reporting a statistically significant meta-analysis of at least four randomized controlled trials in advanced cancer patients using either a fixed-effect or random-effects model. We investigated the impact of switching from fixed-effect to random-effects meta-analysis and of using the recommended Hartung-Knapp method in random-effects meta-analyses. Furthermore, we calculated 95% PIs for all included meta-analyses. 6234 hits were identified, of which 261 statistically significant meta-analyses were included. Our recalculations of these 261 meta-analyses produced statistically significant results in 132 of 138 fixed-effect and 114 of 123 random-effects meta-analyses. When switching to a random-effects model, 19 of 132 fixed-effect meta-analyses (14.4%) were no longer statistically significant. Using the Hartung-Knapp method in random-effects meta-analyses resulted in 34 of 114 non-significant meta-analyses (29.8%). In the full sample (N = 261), the null effect was included by the 95% PI in 195 (74.7%) and the opposite effect (e.g., hazard ratio 0.5, opposite effect 2) in 98 meta-analyses (37.5%). Using the Hartung-Knapp method and PIs substantially influenced the interpretation of many published, statistically significant meta-analyses. We strongly encourage researchers to check if using the Hartung-Knapp method and reporting 95% PIs is appropriate in random-effects meta-analyses. This article is protected by copyright. All rights reserved.

Combining meta-epidemiological study datasets on commercial funding of randomised clinical trials: Database, methods, and descriptive results of the COMFIT study

Randomised trials are often funded by commercial companies and methodological studies support a widely held suspicion that commercial funding may influence trial results and conclusions. However, these studies often have a risk of confounding and reporting bias. The risk of confounding is markedly reduced in meta-epidemiological studies that compare fairly similar trials within meta-analyses, and risk of reporting bias is reduced with access to unpublished data. Therefore, we initiated the COMmercial Funding In Trials (COMFIT) study aimed at investigating the impact of commercial funding on estimated intervention effects in randomised clinical trials based on a consortium of researchers who agreed to share meta-epidemiological study datasets with information on meta-analyses and trials included in meta-epidemiological studies. Here, we describe the COMFIT study, its database, and descriptive results. We included meta-epidemiological studies with published or unpublished data on trial funding source and results or conclusions. We searched five bibliographic databases and other sources. We invited authors of eligible meta-epidemiological studies to join the COMFIT consortium and to share data. The final construction of the COMFIT database involves checking data quality, identifying trial references, harmonising variable categories, and removing non-informative meta-analyses as well as correlated meta-analyses and trial results. We included data from 17 meta-epidemiological studies, covering 728 meta-analyses and 6841 trials. Seven studies (405 meta-analyses, 3272 trials) had not published analyses on the impact of commercial funding, but shared unpublished data on funding source. On this basis, we initiated the construction of a combined database. Once completed, the database will enable comprehensive analyses of the impact of commercial funding on trial results and conclusions with increased statistical power and a markedly reduced risk of confounding and reporting bias.

No evidence found for an association between trial characteristics and treatment effects in randomized trials of testosterone therapy in men: a meta-epidemiological study

OBJECTIVE

The objective of this study was to identify potential trial characteristics associated with reported treatment effect estimates in randomized trials of testosterone therapy in adult men.

STUDY DESIGN AND SETTING

This is a meta-epidemiological study. MEDLINE was searched for meta-analyses of randomized trials of testosterone therapy in men published between 2008 and 2018. Data on trial characteristics were extracted independently by two reviewers. The impact of trial characteristics on reported treatment effects was investigated using a two-step meta-analytic approach.

RESULTS

We identified 132 randomized trials, included in 19 meta-analyses, comprising data from 10,725 participants. None of the investigated design characteristics, including year of publication, sample size, trial registration status, center status, regionality, funding source, and conflict of interest were statistically significantly associated with reported treatment effects of testosterone therapy in men. Although trials rated at high risk of bias overall reported treatment effects that were 21% larger compared with trials rated at low risk of bias overall, the 95% confidence interval included the null (ratio of odds ratio: 0.79, 95% confidence interval: 0.60 to 1.03).

CONCLUSION

The present study found no clear evidence that trial characteristics are associated with treatment effects in randomized trials of testosterone therapy in men. To establish stronger evidence about the treatment effects of testosterone therapy in men, future randomized trials should not only be adequately designed but also transparently reported.

STUDY REGISTRATION

osf.io/x9g6m.

Have We Substantially Underestimated the Impact of Improved Sanitation Coverage on Child Health? A Generalized Additive Model Panel Analysis of Global Data on Child Mortality and Malnutrition

BACKGROUND

Although widely accepted as being one of the most important public health advances of the past hundred years, the contribution that improving sanitation coverage can make to child health is still unclear, especially since the publication of two large studies of sanitation in India which found no effect on child morbidity. We hypothesis that the value of sanitation does not come directly from use of improved sanitation but from improving community coverage. If this is so we further hypothesise that the relationship between sanitation coverage and child health will be non-linear and that most of any health improvement will accrue as sanitation becomes universal.

METHODS

We report a fixed effects panel analysis of country level data using Generalized Additive Models in R. Outcome variables were under 5 childhood mortality, neonatal mortality, under 5 childhood mortality from diarrhoea, proportion of children under 5 with stunting and with underweight. Predictor variables were % coverage by improved sanitation, improved water source, Gross Domestic Product per capita and Health Expenditure per capita. We also identified three studies reporting incidence of diarrhoea in children under five alongside gains in community coverage in improved sanitation.

FINDINGS

For each of the five outcome variables, sanitation coverage was independently associated with the outcome but this association was highly non-linear. Improving sanitation coverage was very strongly associated with under 5 years diarrhoea mortality, under 5years all-cause mortality, and all-cause neonatal mortality. There was a decline as sanitation coverage increased up to about 20% but then no further decline was seen until about 70% (60% for diarrhoea mortality and 80% for neonatal mortality, respectively). The association was less strong for stunting and underweight but a threshold about 50% coverage was also seen. Three large trials of sanitation on diarrhoea morbidity gave results that were similar to what would have been predicted by our model.

CONCLUSIONS

Improving sanitation coverage may be one of the more effective means to reduce childhood mortality, but only if high levels of community coverage are achieved. Studies of the impact of sanitation that focus on the individual's use of improved sanitation as the predictor variable rather than community coverage is likely to severely underestimate the impact of sanitation.

The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses

POLICY POINTS

Currently, there is massive production of unnecessary, misleading, and conflicted systematic reviews and meta-analyses. Instead of promoting evidence-based medicine and health care, these instruments often serve mostly as easily produced publishable units or marketing tools. Suboptimal systematic reviews and meta-analyses can be harmful given the major prestige and influence these types of studies have acquired. The publication of systematic reviews and meta-analyses should be realigned to remove biases and vested interests and to integrate them better with the primary production of evidence.

CONTEXT

Currently, most systematic reviews and meta-analyses are done retrospectively with fragmented published information. This article aims to explore the growth of published systematic reviews and meta-analyses and to estimate how often they are redundant, misleading, or serving conflicted interests.

METHODS

Data included information from PubMed surveys and from empirical evaluations of meta-analyses.

FINDINGS

Publication of systematic reviews and meta-analyses has increased rapidly. In the period January 1, 1986, to December 4, 2015, PubMed tags 266,782 items as "systematic reviews" and 58,611 as "meta-analyses." Annual publications between 1991 and 2014 increased 2,728% for systematic reviews and 2,635% for meta-analyses versus only 153% for all PubMed-indexed items. Currently, probably more systematic reviews of trials than new randomized trials are published annually. Most topics addressed by meta-analyses of randomized trials have overlapping, redundant meta-analyses; same-topic meta-analyses may exceed 20 sometimes. Some fields produce massive numbers of meta-analyses; for example, 185 meta-analyses of antidepressants for depression were published between 2007 and 2014. These meta-analyses are often produced either by industry employees or by authors with industry ties and results are aligned with sponsor interests. China has rapidly become the most prolific producer of English-language, PubMed-indexed meta-analyses. The most massive presence of Chinese meta-analyses is on genetic associations (63% of global production in 2014), where almost all results are misleading since they combine fragmented information from mostly abandoned era of candidate genes. Furthermore, many contracting companies working on evidence synthesis receive industry contracts to produce meta-analyses, many of which probably remain unpublished. Many other meta-analyses have serious flaws. Of the remaining, most have weak or insufficient evidence to inform decision making. Few systematic reviews and meta-analyses are both non-misleading and useful.

CONCLUSIONS

The production of systematic reviews and meta-analyses has reached epidemic proportions. Possibly, the large majority of produced systematic reviews and meta-analyses are unnecessary, misleading, and/or conflicted.

Comparison of intervention effects in split-mouth and parallel-arm randomized controlled trials: a meta-epidemiological study

Background

Split-mouth randomized controlled trials (RCTs) are popular in oral health research. Meta-analyses frequently include trials of both split-mouth and parallel-arm designs to derive combined intervention effects. However, carry-over effects may induce bias in split- mouth RCTs. We aimed to assess whether intervention effect estimates differ between split- mouth and parallel-arm RCTs investigating the same questions.

Methods

We performed a meta-epidemiological study. We systematically reviewed meta- analyses including both split-mouth and parallel-arm RCTs with binary or continuous outcomes published up to February 2013. Two independent authors selected studies and extracted data. We used a two-step approach to quantify the differences between split-mouth and parallel-arm RCTs: for each meta-analysis. First, we derived ratios of odds ratios (ROR) for dichotomous data and differences in standardized mean differences (∆SMD) for continuous data; second, we pooled RORs or ∆SMDs across meta-analyses by random-effects meta-analysis models.

Results

We selected 18 systematic reviews, for 15 meta-analyses with binary outcomes (28 split-mouth and 28 parallel-arm RCTs) and 19 meta-analyses with continuous outcomes (28 split-mouth and 28 parallel-arm RCTs). Effect estimates did not differ between split-mouth and parallel-arm RCTs (mean ROR, 0.96, 95% confidence interval 0.52–1.80; mean ∆SMD, 0.08, -0.14–0.30).

Conclusions

Our study did not provide sufficient evidence for a difference in intervention effect estimates derived from split-mouth and parallel-arm RCTs. Authors should consider including split-mouth RCTs in their meta-analyses with suitable and appropriate analysis.

Evidence synthesis for decision making 3: heterogeneity--subgroups, meta-regression, bias, and bias-adjustment

In meta-analysis, between-study heterogeneity indicates the presence of effect-modifiers and has implications for the interpretation of results in cost-effectiveness analysis and decision making. A distinction is usually made between true variability in treatment effects due to variation in patient populations or settings and biases related to the way in which trials were conducted. Variability in relative treatment effects threatens the external validity of trial evidence and limits the ability to generalize from the results; imperfections in trial conduct represent threats to internal validity. We provide guidance on methods for meta-regression and bias-adjustment, in pairwise and network meta-analysis (including indirect comparisons), using illustrative examples. We argue that the predictive distribution of a treatment effect in a "new" trial may, in many cases, be more relevant to decision making than the distribution of the mean effect. Investigators should consider the relative contribution of true variability and random variation due to biases when considering their response to heterogeneity. In network meta-analyses, various types of meta-regression models are possible when trial-level effect-modifying covariates are present or suspected. We argue that a model with a single interaction term is the one most likely to be useful in a decision-making context. Illustrative examples of Bayesian meta-regression against a continuous covariate and meta-regression against "baseline" risk are provided. Annotated WinBUGS code is set out in an appendix.