"The challenge of meta-analysis": discussion. Indications and contra-indications for meta-analysis

Twenty years of network meta-analysis: Continuing controversies and recent developments

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

The Chiropractic Care of Infants with Breastfeeding Difficulties

OBJECTIVE

Chiropractors have long advocated on the benefits of breastfeeding and given the realized and potential role of chiropractors in the care of infants with breastfeeding difficulties, we performed this review of the literature on the subject to inform clinical practice.

METHODS

For this article, we searched Pubmed [1966-2013], Manual, Alternative and Natural Therapy Index System (MANTIS) [1964-2013] and Index to Chiropractic Literature [1984-2013] for the relevant literature. The search terms utilized "breastfeeding", "breast feeding", "breastfeeding difficulties", "breastfeeding difficulty", "TMJ dysfunction", "temporomandibular joint", "birth trauma" and "infants", in the appropriate Boolean combinations. We also examined non-peer-reviewed articles as revealed by Index to Chiropractic Literature and secondary analysis of references. Inclusion criteria for review included breastfeeding difficulties regardless of peer-review and written in the English language.

RESULTS

A total of 24 articles met our inclusion criteria. These consisted of 8 case reports, 2 case series, and 3 cohort studies. We were also able to identify 6 manuscripts (5 case reports and a case series) that involved breastfeeding difficulties as a secondary complaint. Our findings reveal a theoretical and clinical framework based on the detection of spinal and extraspinal subluxations involving the cervico-cranio-mandibular complex and assessment of the infant while breastfeeding.

CONCLUSIONS

Chiropractors care of infants with breastfeeding difficulties by addressing spinal and extraspinal subluxations involving the cervico-cranio-mandibular complex.

Clinical effectiveness of chin cup treatment for the management of Class III malocclusion in pre-pubertal patients: a systematic review and meta-analysis

Background

Chin cup is regarded as the oldest orthodontic appliance for the management of Class III malocclusion. To assess its clinical effectiveness in pre-pubertal patients, a meta-analysis on specific cephalometric values is attempted.

Methods

Detailed electronic and hand searches with no restrictions were performed up to July 2014. Only randomized controlled trials (RCTs) and cohort studies, i.e. prospective controlled trials (pCCTs) and (retrospective) observational studies (OS), were included. Analyses were performed by calculating the standard difference in means and the corresponding 95% confidence intervals, using the random effects model. Data heterogeneity and risk of bias assessment of the included studies were also performed. Study selection, data extraction and risk of bias assessment were performed twice. The level of significance was set at P ≤ 0.05 for all tests, except for heterogeneity (P ≤ 0.1).

Results

Seven treated groups from five studies (no RCTs, four pCCTs, one OS) were eligible for inclusion, assessing only the short-term occipital pull chin cup effects. In total, 120 treated patients (mean age: 8.5 to 11 years) compared with 64 untreated individuals (mean age: 7.3 to 9.89 years) were assessed by means of 13 cephalometric variables. The overall quality of these studies was low to medium. In comparison to untreated individuals, the SNB and gonial angles decreased significantly following chin cup use, whereas ANB, Wits appraisal, SN-ML, N-Me and overjet increased. For the rest of the variables, no statistically significant differences were detected.

Conclusions

Although the occipital chin cup affects significantly a number of skeletal and dentoalveolar cephalometric variables, indicating an overall positive effect for the treatment of Class III malocclusion, data heterogeneity and between-studies variance impose precaution in the interpretation of the results.

Electronic supplementary material

The online version of this article (doi:10.1186/s40510-014-0062-9) contains supplementary material, which is available to authorized users.

Critical evaluation of mixed treatment comparison meta-analyses using examples assessing antidepressants and opioid detoxification treatments

Comparing multiple treatment options using meta-analytical methods requires complex statistical methods called mixed treatment comparisons (MTCs). Such methods offer the possibility to summarize data from many clinical trials comparing the different available options. However, those methods are based on a number of assumptions and inherent difficulties that are discussed and illustrated with examples from the psychiatric literature to help readers to understand the strengths and weaknesses of these methods. This review will help enable readers to critically appraise the methodology and results of publications that use MTCs.

Do unblinded assessors bias muscle strength outcomes in randomized controlled trials of progressive resistance strength training in older adults?

OBJECTIVE

Knowledge of treatment assignment and failing to analyze results by randomized treatment groups--an intention-to-treat analysis--may cause bias in the treatment effect estimate in randomized controlled trials. This study was undertaken to determine the difference in lower limb muscle strength measured by blinded vs. by unblinded outcome assessors in 73 progressive resistance strength training trials conducted in older adults.

DESIGN

Retrospective analysis of randomized controlled trials published before 2007.

RESULTS

Meta-regression analyses showed that trials that used blinded assessors (n = 18) tend to report smaller effect sizes than do those that used unblinded assessors (n = 55), with a difference of -0.80 (95% confidence interval, -1.35 to -0.25). This result still holds even after adjusting for the use of an intention-to-treat analysis, with an adjusted difference of -0.65 (95% confidence interval, -1.26 to -0.04). The reported effects were exaggerated in trials that used unblinded assessors.

CONCLUSIONS

This study suggests that assessor blinding is important and is a safeguard to the internal validity of exercise trials in older adults.

Timing of surgery in cauda equina syndrome with urinary retention: meta-analysis of observational studies

Object

The authors performed exploratory meta-analyses of observational cohort studies, evidence level III, examining whether earlier surgery makes a difference in outcome in terms of urinary function once cauda equina syndrome (CES) from a herniated lumbar disc has progressed to urinary retention (CESR).

Methods

Literature search identified 27 studies of CESR patients with clear definition of surgical timing. Relative risk (RR) could not be calculated in 11 studies, leaving 16 for meta-analysis. Urinary retention related to surgical timing at 5 breakpoints: 12, 24, 36, 48, or 72 hours. Urinary outcome was classified as Normal, Fair, or Poor. Meta-analysis was performed for “Event = Fair/Poor” or “Event = Poor.” Eight studies allowed separation into CESR and incomplete CES (CESI), and 5 of these had enough data for meta-analysis to compare CESR and CESI. A random effects meta-analysis model was used because of heterogeneity across the studies. A best-evidence synthesis was performed for the 4 largest studies that had 24- and 48-hour breakpoints.

Results

For “Event = Fair/Poor,” meta-analyses using the 5 breakpoints predicted a more likely Fair/Poor outcome for later surgery (RR range 1.77–2.19). The RR for later timing of surgery was statistically significant for 24-and 72-hour breakpoints and was elevated but not statistically significant for the other 3. For “Event = Poor,” the RR range was 1.09–5.82, statistically significant for the 36 hour breakpoint only. Meta-analysis comparing CESR patients with CESI patients predicted a Fair/Poor result for CESR (RR 2.58, 95% confidence interval 0.59–11.31). The best-evidence synthesis did not disclose reasons for differences in the results of the 4 studies.

Conclusions

This study supports early surgery for CES and indicates that CESR and CESI cases should not be analyzed together.

Combination of direct and indirect evidence in mixed treatment comparisons

Mixed treatment comparison (MTC) meta-analysis is a generalization of standard pairwise meta-analysis for A vs B trials, to data structures that include, for example, A vs B, B vs C, and A vs C trials. There are two roles for MTC: one is to strengthen inference concerning the relative efficacy of two treatments, by including both 'direct' and 'indirect' comparisons. The other is to facilitate simultaneous inference regarding all treatments, in order for example to select the best treatment. In this paper, we present a range of Bayesian hierarchical models using the Markov chain Monte Carlo software WinBUGS. These are multivariate random effects models that allow for variation in true treatment effects across trials. We consider models where the between-trials variance is homogeneous across treatment comparisons as well as heterogeneous variance models. We also compare models with fixed (unconstrained) baseline study effects with models with random baselines drawn from a common distribution. These models are applied to an illustrative data set and posterior parameter distributions are compared. We discuss model critique and model selection, illustrating the role of Bayesian deviance analysis, and node-based model criticism. The assumptions underlying the MTC models and their parameterization are also discussed.

Proportional odds ratio model for comparison of diagnostic tests in meta-analysis

BACKGROUND

Consider a meta-analysis where a 'head-to-head' comparison of diagnostic tests for a disease of interest is intended. Assume there are two or more tests available for the disease, where each test has been studied in one or more papers. Some of the papers may have studied more than one test, hence the results are not independent. Also the collection of tests studied may change from one paper to the other, hence incomplete matched groups.

METHODS

We propose a model, the proportional odds ratio (POR) model, which makes no assumptions about the shape of ORp, a baseline function capturing the way OR changes across papers. The POR model does not assume homogeneity of ORs, but merely specifies a relationship between the ORs of the two tests. One may expand the domain of the POR model to cover dependent studies, multiple outcomes, multiple thresholds, multi-category or continuous tests, and individual-level data.

RESULTS

In the paper we demonstrate how to formulate the model for a few real examples, and how to use widely available or popular statistical software (like SAS, R or S-Plus, and Stata) to fit the models, and estimate the discrimination accuracy of tests. Furthermore, we provide code for converting ORs into other measures of test performance like predictive values, post-test probabilities, and likelihood ratios, under mild conditions. Also we provide code to convert numerical results into graphical ones, like forest plots, heterogeneous ROC curves, and post test probability difference graphs.

CONCLUSIONS

The flexibility of POR model, coupled with ease with which it can be estimated in familiar software, suits the daily practice of meta-analysis and improves clinical decision-making.

Efficacy of spinal manipulation and mobilization for low back pain and neck pain: a systematic review and best evidence synthesis

BACKGROUND CONTEXT

Despite the many published randomized clinical trials (RCTs), a substantial number of reviews and several national clinical guidelines, much controversy still remains regarding the evidence for or against efficacy of spinal manipulation for low back pain and neck pain.

PURPOSE

To reassess the efficacy of spinal manipulative therapy (SMT) and mobilization (MOB) for the management of low back pain (LBP) and neck pain (NP), with special attention to applying more stringent criteria for study admissibility into evidence and for isolating the effect of SMT and/or MOB.

STUDY DESIGN

RCTs including 10 or more subjects per group receiving SMT or MOB and using patient-oriented primary outcome measures (eg, patient-rated pain, disability, global improvement and recovery time).

METHODS

Articles in English, Danish, Swedish, Norwegian and Dutch reporting on randomized trials were identified by a comprehensive search of computerized and bibliographic literature databases up to the end of 2002. Two reviewers independently abstracted data and assessed study quality according to eight explicit criteria. A best evidence synthesis incorporating explicit, detailed information about outcome measures and interventions was used to evaluate treatment efficacy. The strength of evidence was assessed by a classification system that incorporated study validity and statistical significance of study results. Sixty-nine RCTs met the study selection criteria and were reviewed and assigned validity scores varying from 6 to 81 on a scale of 0 to 100. Forty-three RCTs met the admissibility criteria for evidence.

RESULTS

Acute LBP: There is moderate evidence that SMT provides more short-term pain relief than MOB and detuned diathermy, and limited evidence of faster recovery than a commonly used physical therapy treatment strategy. Chronic LBP: There is moderate evidence that SMT has an effect similar to an efficacious prescription nonsteroidal anti-inflammatory drug, SMT/MOB is effective in the short term when compared with placebo and general practitioner care, and in the long term compared to physical therapy. There is limited to moderate evidence that SMT is better than physical therapy and home back exercise in both the short and long term. There is limited evidence that SMT is superior to sham SMT in the short term and superior to chemonucleolysis for disc herniation in the short term. However, there is also limited evidence that MOB is inferior to back exercise after disc herniation surgery. Mix of acute and chronic LBP: SMT/MOB provides either similar or better pain outcomes in the short and long term when compared with placebo and with other treatments, such as McKenzie therapy, medical care, management by physical therapists, soft tissue treatment and back school. Acute NP: There are few studies, and the evidence is currently inconclusive. Chronic NP: There is moderate evidence that SMT/MOB is superior to general practitioner management for short-term pain reduction but that SMT offers at most similar pain relief to high-technology rehabilitative exercise in the short and long term. Mix of acute and chronic NP: The overall evidence is not clear. There is moderate evidence that MOB is superior to physical therapy and family physician care, and similar to SMT in both the short and long term. There is limited evidence that SMT, in both the short and long term, is inferior to physical therapy.

CONCLUSIONS

Our data synthesis suggests that recommendations can be made with some confidence regarding the use of SMT and/or MOB as a viable option for the treatment of both low back pain and NP. There have been few high-quality trials distinguishing between acute and chronic patients, and most are limited to shorter-term follow-up. Future trials should examine well-defined subgroups of patients, further address the value of SMT and MOB for acute patients, establish optimal number of treatment visits and consider the cost-effectiveness of care.

Effektivität experimenteller und rehabilitativer Interventionen bei visuellem Neglect - Eine Metaanalyse

Zusammenfassung: Präsentiert wird eine Metaanalyse zur Wirksamkeit neuropsychologischer Interventionen bei visuellem Hemineglect. Es wurden die publizierten Outcome-Ergebnisse Experimental- und Rehabilitationsstudien zur Modifikation und Modulation des Neglectsyndroms quantitativ integriert. 133 identifizierte Studien wurden in sechs Klassen eingeteilt. Studien-, Interventions- und Patientenmerkmale dienten zur Bestimmung moderierender Einflüsse, über Sensitivitätsanalysen wurde die Robustheit der Ergebnisse abgeschätzt. Die mittlere Gesamteffektivität der Interventionen liegt um 0.8 Standardeinheiten Symptomverbesserung gegenüber dem Prä-Niveau bei Patienten mit chronischem Neglect. Spontanremissionseffekte sind aufgrund der durchschnittlichen Erkrankungsdauer als gering zu veranschlagen und konnten empirisch nicht nachgewiesen werden. Explorationstrainings, personale Intervention und Hinweisreiz-Verfahren stellen sich als überdurchschnittlich effektiv und signifikant effektiver als bloße Reduktion der Umweltanforderungen heraus.

Isolated single umbilical artery anomaly and the risk for congenital malformations: a meta-analysis

BACKGROUND/PURPOSE

Single umbilical artery (SUA) is an associated finding in many chromosomal abnormalities and congenital malformations. However, SUAcan also be seen as an isolated finding. The extent of diagnostic investigation and follow-up needed in infants with SUA as an isolated finding has long been debated. In this study the authors examined this issue by an analysis of the published papers concerning SUA anomaly.

METHODS

A meta-analysis of 37 studies related to SUA published over a 40-year period was performed. Eleven of 37 studies were considered group 1 because the diagnosis of SUA was made using specimens obtained from early abortuses, fetal deaths, or autopsies. The remaining 26 studies were considered group 2 because the diagnosis of SUA was made from examining placental and umbilical cord specimens obtained from live-born infants or by examination of the umbilicus soon after birth.

RESULTS

The mean (and 95% CI) SUA incidence was 2.13% (1.92%, 2.34%) in group 1, and 0.55% (0.44%, 0.66%) in group 2. The mean incidence of congenital malformation associated with SUA was 66.3% (67.7%, 70.8%) in group 1, and 27% (21.6%, 32.3%) in group 2; these differences were significant (P< .05). In 7 of 26 group 2 studies, additional urologic investigations were performed in 204 infants who had an isolated SUA anomaly. Thirty-three of these 204 infants (16.2%, 95% CI, 7.7%, 25.6%) had some form of renal anomaly, but in 18 of 33 (54.5%) the anomalies were minor or self limiting. A cost-versus-benefit analysis showed that 14 isolated SUA cases need to be investigated to detect one major renal malformation; however, the value of detecting them early remained unclear because most of the anomalies would be apparent with good pediatric follow-up.

CONCLUSIONS

When SUA is an isolated anomaly in an otherwise healthy infant, there is a slight increase in the risk for renal malformations. This association is of similar magnitude to the coexistence of another anomaly with the presence of one anomaly. Most renal anomalies in isolated SUA cases are minor and self limiting, and even major anomalies are no different from those that are diagnosed during routine pediatric follow-up in otherwise healthy infants. Therefore, unless additional risk factors for malformations exist, the current data do not justify extensive urologic radiographic investigations in asymptomatic newborns having an isolated SUA anomaly.

Overcoming the limitations of current meta-analysis of randomised controlled trials

For a meta-analysis to give definitive information, it should meet at least the minimum standards that would be expected of a well-designed, adequately powered, and carefully conducted randomised controlled trial. These minimum standards include both qualitative characteristics--a prospective protocol, comparable definitions of key outcomes, quality control of data, and inclusion of all patients from all trials in the final analysis--and quantitative standards--an assessment of whether the total sample is large enough to provide reliable results and the use of appropriate statistical monitoring guidelines to indicate when the results of the accumulating data of a meta-analysis are conclusive. We believe that rigorous meta-analyses undertaken according to these principles will lead to more reliable evidence about the efficacy and safety of interventions than either retrospective meta-analysis or individual trials.

Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away

This paper examines how threats to the validity of meta-analysis have been dealt with by clinical researchers employing this approach to literature review. Three validity threats were identified--mixing of dissimilar studies, publication bias, and inclusion of poor quality studies. Approaches to addressing these threats were evaluated for their effectiveness and popularity by surveying 32 published meta-analyses in clinical psychology. Distrust of meta-analysis, however, was found to transcend these validity threats. Other explanations for why this popular research strategy continues to receive widespread criticism were considered. Suggestions were made for how meta-analysis might better address these concerns.

A new challenge in clinical research in childhood ALL: the prospective meta-analysis strategy for intergroup collaboration

We consider the problems arising in clinical research on childhood acute lymphoblastic leukemia (ALL). Given the therapeutic progress achieved over the last few decades, any improvement in the outcome for the majority of children with ALL is difficult to assess with the usual size trials. Furthermore, the progress in genetics and molecular biology has now led to the identification of subgroups of children, typically with rare characteristics, for whom new treatments still await evaluation. For both these aspects of clinical research, there is an increasing need for international intergroup cooperation. After a discussion on the role of retrospective meta-analysis and randomized controlled trials in ALL research, we suggest that intergroup studies could be made more feasible, but still scientifically rigorous, by adopting a strategy of prospective meta-analysis. This strategy can be described as follows: i) different groups prospectively plan to ask the same randomized question within their protocols which may differ in other aspects, and to pool their data in order to evaluate treatment effect; ii) the management of the study can be de-centralized, by allowing each group to be responsible for conducting its own protocol. We would like to stimulate the debate on the methodological and practical aspects of research perspectives in ALL (and in pediatric oncology).

The relation between treatment benefit and underlying risk in meta-analysis

In meta-analyses of clinical trials comparing a treated group with a control group it has been common to ask whether the treatment benefit varies according to the underlying risk of the patients in the different trials, with the hope of defining which patients would benefit most and which least from medical interventions. The usual analysis used to investigate this issue, however, which uses the observed proportions of events in the control groups of the trials as a measure of the underlying risk, is flawed and produces seriously misleading results. This arises through a bias due to regression to the mean and will be particularly acute in meta-analyses which include some small trials or in which the variability in the true underlying risks across trials is small. Approaches which previously have been thought to be more appropriate are to substitute the average proportion of events in the control and treated groups as the measure of underlying risk or to plot the proportion of events in the treated group against that in the control group (L'Abbé plot). However, these are still subject to bias in most circumstances. Because of the potentially seriously flawed conclusions that can result from such analyses, they should be replaced either by statistically appropriate (but more complex) approaches or, preferably, by analyses which investigate the dependence of the treatment effect on measured baseline characteristics of the patients in each trial.