Adjusting trial results for biases in meta-analysis: combining data-based evidence on bias with detailed trial assessment

Flaws in the conduct of randomized trials can lead to biased estimation of the intervention effect. Methods for adjustment of within-trial biases in meta-analysis include the use of empirical evidence from an external collection of meta-analyses, and the use of expert opinion informed by the assessment of detailed trial information. Our aim is to present methods to combine these two approaches to gain the advantages of both. We make use of the risk of bias information that is routinely available in Cochrane reviews, by obtaining empirical distributions for the bias associated with particular bias profiles (combinations of risk of bias judgements). We propose three methods: a formal combination of empirical evidence and opinion in a Bayesian analysis; asking experts to give an opinion on bias informed by both summary trial information and a bias distribution from the empirical evidence, either numerically or by selecting areas of the empirical distribution. The methods are demonstrated through application to two example binary outcome meta-analyses. Bias distributions based on opinion informed by trial information alone were most dispersed on average, and those based on opinions obtained by selecting areas of the empirical distribution were narrowest. Although the three methods for combining empirical evidence with opinion vary in ease and speed of implementation, they yielded similar results in the two examples.

The effects of changes to the built environment on the mental health and well-being of adults: Systematic review

There is increasing interest in the influence of place on health, and the need to distinguish between environmental and individual level factors. For environmental-level factors, current evidence tends to show associations through cross-sectional and uncontrolled longitudinal analyses rather than through more robust study designs that can provide stronger causal evidence. We restricted this systematic review to randomised (or cluster) randomised controlled trials and controlled before-and-after studies of changes to the built environment. Date of search was December 2016. We identified 14 studies. No evidence was found of an effect on mental health from 'urban regeneration' and 'improving green infrastructure' studies. Beneficial effects on quality-of-life outcomes from 'improving green infrastructure' were found in two studies. One 'improving green infrastructure' study reported an improvement in social isolation. Risk-of-bias assessment indicated robust data from only four studies. Overall, evidence for the impact of built environment interventions on mental health and quality-of-life is weak. Future research requires more robust study designs and interdisciplinary research involving public health, planning and urban design experts.

Label-invariant models for the analysis of meta-epidemiological data

Rich meta‐epidemiological data sets have been collected to explore associations between intervention effect estimates and study‐level characteristics. Welton et al proposed models for the analysis of meta‐epidemiological data, but these models are restrictive because they force heterogeneity among studies with a particular characteristic to be at least as large as that among studies without the characteristic. In this paper we present alternative models that are invariant to the labels defining the 2 categories of studies. To exemplify the methods, we use a collection of meta‐analyses in which the Cochrane Risk of Bias tool has been implemented. We first investigate the influence of small trial sample sizes (less than 100 participants), before investigating the influence of multiple methodological flaws (inadequate or unclear sequence generation, allocation concealment, and blinding). We fit both the Welton et al model and our proposed label‐invariant model and compare the results. Estimates of mean bias associated with the trial characteristics and of between‐trial variances are not very sensitive to the choice of model. Results from fitting a univariable model show that heterogeneity variance is, on average, 88% greater among trials with less than 100 participants. On the basis of a multivariable model, heterogeneity variance is, on average, 25% greater among trials with inadequate/unclear sequence generation, 51% greater among trials with inadequate/unclear blinding, and 23% lower among trials with inadequate/unclear allocation concealment, although the 95% intervals for these ratios are very wide. Our proposed label‐invariant models for meta‐epidemiological data analysis facilitate investigations of between‐study heterogeneity attributable to certain study characteristics.

Systematic review and critical appraisal of the impact of acellular dermal matrix use on the outcomes of implant-based breast reconstruction

BACKGROUND

Acellular dermal matrix (ADM) may improve outcomes in implant-based breast reconstruction (IBBR). The aim of this study was critically to appraise and evaluate the current evidence for ADM-assisted IBBR.

METHODS

Comprehensive electronic searches identified complete papers published in English between January 2000 and August 2013, reporting any outcome of ADM-assisted IBBR. All systematic reviews, randomized clinical trials (RCTs) and non-randomized studies (NRSs) with more than 20 ADM recipients were included. Studies were critically appraised using AMSTAR for systematic reviews, the Cochrane risk-of-bias tool for RCTs and its adaptation for NRSs. Characteristics and results of identified studies were summarized.

RESULTS

A total of 69 papers (8 systematic reviews, 1 RCT, 40 comparative studies and 20 case series) were identified, all of which were considered at high risk of bias, mostly due to patient selection and selective outcome reporting. The median ADM group sample size was 51.0 (i.q.r. 33.0-127.0). Most studies were single-centre (54), and they were often single-surgeon (16). ADM was most commonly used for immediate (40) two-stage IBBR (36) using human ADM (47), with few studies evaluating ADM-assisted single-stage procedures (10). All reported clinical outcomes (for example implant loss) and more than half of the papers (33) assessed process outcomes, but few evaluated cosmesis (16) or patient-reported outcomes (10). Heterogeneity between study design and, especially, outcome measurement precluded meaningful data synthesis.

CONCLUSION

Current evidence for the value of ADMs in IBBR is limited. Use in practice should therefore be considered experimental, and evaluation within registries or well designed and conducted studies, ideally RCTs, is recommended to prevent widespread adoption of a potentially inferior intervention.

Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies

BACKGROUND

The design of randomised controlled trials (RCTs) should incorporate characteristics (such as concealment of randomised allocation and blinding of participants and personnel) that avoid biases resulting from lack of comparability of the intervention and control groups. Empirical evidence suggests that the absence of such characteristics leads to biased intervention effect estimates, but the findings of different studies are not consistent.

OBJECTIVES

To examine the influence of unclear or inadequate random sequence generation and allocation concealment, and unclear or absent double blinding, on intervention effect estimates and between-trial heterogeneity, and whether or not these influences vary with type of clinical area, intervention, comparison and outcome measure.

DATA SOURCES AND METHODS

Data were combined from seven contributing meta-epidemiological studies (collections of meta-analyses in which trial characteristics are assessed and results recorded). The resulting database was used to identify and remove overlapping meta-analyses. Outcomes were coded such that odds ratios < 1 correspond to beneficial intervention effects. Outcome measures were classified as mortality, other objective or subjective. We examined agreement between assessments of trial characteristics in trials assessed in more than one contributing study. We used hierarchical Bayesian bias models to estimate the effect of trial characteristics on average bias [quantified as ratios of odds ratios (RORs) with 95% credible intervals (CrIs) comparing trials with and without a characteristic] and in increasing between-trial heterogeneity.

RESULTS

The analysis data set contained 1973 trials included in 234 meta-analyses. Median kappa statistics for agreement between assessments of trial characteristics were: sequence generation 0.60, allocation concealment 0.58 and blinding 0.87. Intervention effect estimates were exaggerated by an average 11% in trials with inadequate or unclear (compared with adequate) sequence generation (ROR 0.89, 95% CrI 0.82 to 0.96); between-trial heterogeneity was higher among such trials. Bias associated with inadequate or unclear sequence generation was greatest for subjective outcomes (ROR 0.83, 95% CrI 0.74 to 0.94) and the increase in heterogeneity was greatest for such outcomes [standard deviation (SD) 0.20, 95% CrI 0.03 to 0.32]. The effect of inadequate or unclear (compared with adequate) allocation concealment was greatest among meta-analyses with a subjectively assessed outcome intervention effect (ROR 0.85, 95% CrI 0.75 to 0.95), and the increase in between-trial heterogeneity was also greatest for such outcomes (SD 0.20, 95% CrI 0.02 to 0.33). Lack of, or unclear, double blinding (compared with double blinding) was associated with an average 13% exaggeration of intervention effects (ROR 0.87, 95% CrI 0.79 to 0.96), and between-trial heterogeneity was increased for such studies (SD 0.14, 95% CrI 0.02 to 0.30). Average bias (ROR 0.78, 95% CrI 0.65 to 0.92) and between-trial heterogeneity (SD 0.37, 95% CrI 0.19 to 0.53) were greatest for meta-analyses assessing subjective outcomes. Among meta-analyses with subjectively assessed outcomes, the effect of lack of blinding appeared greater than the effect of inadequate or unclear sequence generation or allocation concealment.

CONCLUSIONS

Bias associated with specific reported study design characteristics leads to exaggeration of beneficial intervention effect estimates and increases in between-trial heterogeneity. For each of the three characteristics assessed, these effects were greatest for subjectively assessed outcomes. Assessments of the risk of bias in RCTs should account for these findings. Further research is needed to understand the effects of attrition bias, as well as the relative importance of blinding of patients, care-givers and outcome assessors, and thus separate the effects of performance and detection bias.

FUNDING

National Institute for Health Research Health Technology Assessment programme.