Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis

Visualizing and diagnosing spillover within randomized concurrent controlled trials through the application of diagnostic test assessment methods

BACKGROUND

Spillover of effect, whether positive or negative, from intervention to control group patients invalidates the Stable Unit Treatment Variable Assumption (SUTVA). SUTVA is critical to valid causal inference from randomized concurrent controlled trials (RCCT). Spillover of infection prevention is an important population level effect mediating herd immunity. This herd effect, being additional to any individual level effect, is subsumed within the overall effect size (ES) estimate derived by contrast-based techniques from RCCT's. This herd effect would manifest only as increased dispersion among the control group infection incidence rates above background.

METHODS AND RESULTS

The objective here is to explore aspects of spillover and how this might be visualized and diagnosed. I use, for illustration, data from 190 RCCT's abstracted in 13 Cochrane reviews of various antimicrobial versus non-antimicrobial based interventions to prevent pneumonia in ICU patients. Spillover has long been postulated in this context. Arm-based techniques enable three approaches to identify increased dispersion, not available from contrast-based techniques, which enable the diagnosis of spillover within antimicrobial versus non-antimicrobial based infection prevention RCCT's. These three approaches are benchmarking the pneumonia incidence rates versus a clinically relevant range, comparing the dispersion in pneumonia incidence among the control versus the intervention groups and thirdly, visualizing the incidence dispersion within summary receiver operator characteristic (SROC) plots. By these criteria there is harmful spillover effects to concurrent control group patients.

CONCLUSIONS

Arm-based versus contrast-based techniques lead to contrary inferences from the aggregated RCCT's of antimicrobial based interventions despite similar summary ES estimates. Moreover, the inferred relationship between underlying control group risk and ES is 'flipped'.

A systematic survey of 200 systematic reviews with network meta-analysis (published 2020-2021) reveals that few reviews report structured evidence summaries

OBJECTIVES

To map whether and how systematic reviews (SRs) with network meta-analysis (NMA) use presentation formats to report (a) structured evidence summaries - here defined as reporting of effects estimates in absolute effects with certainty ratings and with a method to rate interventions across one or more outcome(s) - and (b) NMA results in general.

STUDY DESIGN AND SETTING

We conducted a systematic survey, searching MEDLINE (Ovid) for SRs with NMA published between January 1, 2020, and December 31, 2021. We planned to include a random sample of publications, with predefined mechanisms in place for saturation, and included SRs that met prespecified quality criteria and extracted data on presentation formats that reported: (a) estimates of effects, (b) certainty of the evidence, or (c) rating of interventions.

RESULTS

The 200 eligible SRs, from 158 unique Journals, utilized 1133 presentation formats. We found structured evidence summaries in 10 publications (5.0%), with 3 (1.5%) reporting structured evidence summaries across all outcomes, including benefits and harms. Sixteen of the 133 SRs (11.7%) reporting dichotomous outcomes included estimates of absolute effects. Seventy-six SRs (38.0%) reported both benefits and harms and 26 SRs (13.0%) reported certainty ratings in presentation formats, 20 (76.9%) used Grading of Recommendations Assessment, Development and Evaluation and 6 (23.1%) used Confidence In Network Meta-analysis. Surface Under the Cumulative Ranking Curve was the most common method to rate interventions (69 SRs, 34.5%). NMA results were most often reported using forest plots (108 SRs, 54.0%) and league tables (93 SRs, 46.5%).

CONCLUSION

Most SRs with NMA do not report structured evidence summaries and only rarely do such summaries include reporting of both benefits and harms; those that do offer effective user-friendly communication and provide models for optimal NMA presentation practice.

Heterogeneity in meta-analysis: a comprehensive overview

In recent years, meta-analysis has evolved to a critically important field of Statistics, and has significant applications in Medicine and Health Sciences. In this work we briefly present existing methodologies to conduct meta-analysis along with any discussion and recent developments accompanying them. Undoubtedly, studies brought together in a systematic review will differ in one way or another. This yields a considerable amount of variability, any kind of which may be termed heterogeneity. To this end, reports of meta-analyses commonly present a statistical test of heterogeneity when attempting to establish whether the included studies are indeed similar in terms of the reported output or not. We intend to provide an overview of the topic, discuss the potential sources of heterogeneity commonly met in the literature and provide useful guidelines on how to address this issue and to detect heterogeneity. Moreover, we review the recent developments in the Bayesian approach along with the various graphical tools and statistical software that are currently available to the analyst. In addition, we discuss sensitivity analysis issues and other approaches of understanding the causes of heterogeneity. Finally, we explore heterogeneity in meta-analysis for time to event data in a nutshell, pointing out its unique characteristics.

Laser therapy decreases oral leukoplakia recurrence and boosts patient comfort: a network meta-analysis and systematic review

BACKGROUND

Oral leukoplakia (OLK) is a prevalent precancerous lesion with limited non-pharmacological treatment options. Surgery and various lasers are the mainstay of treatment; however, their relative efficacy and optimal choice remain unclear. This first network meta-analysis compared the effects of different lasers and surgical excision on post-treatment recurrence and comfort in OLK patients.

METHODS

We searched four databases for relevant randomized controlled trials (RCTs) up to April 2023. The primary outcome was post-treatment recurrence, and secondary outcomes included intraoperative hemorrhage and postoperative pain scores. The Cochrane Risk of Bias tool was used to assess the study quality. Meta-analysis and network meta-analysis were employed to determine efficacy and identify the optimal intervention.

RESULTS

A total of 11 RCTs including 917 patients and 1138 lesions were included. Er,Cr:YSGG laser treatment showed significantly lower recurrence rates compared to CO2 laser (OR: 0.04; 95% CI: 0.01-0.18), CO2 laser with margin extension (OR: 0.06; 95% CI: 0.01-0.60), Er:YAG laser (OR: 0.10; 95% CI: 0.03-0.37), electrocautery (OR: 0.03; 95% CI: 0.00-0.18), and standard care (OR: 0.08; 95% CI: 0.02-0.33). Er,Cr:YSGG laser also ranked the best for reducing recurrence, followed by standard care and CO2 laser combined with photodynamic therapy (PDT). Er:YAG and Er:Cr:YSGG lasers minimized bleeding and pain, respectively. None of the interventions caused severe adverse effects.

CONCLUSION

For non-homogeneous OLK, Er:YAG, Er:Cr:YSGG, and CO2 laser combined with PDT offer promising alternatives to surgical excision, potentially reducing recurrence and improving patient comfort. Further high-quality RCTs are necessary to confirm these findings and determine the optimal laser-PDT combination for OLK treatment.

Lower Rates of Hypocalcemia Following Near-Infrared Autofluorescence Use in Thyroidectomy: A Meta-Analysis of RCTs

BACKGROUND

Iatrogenic injury of the parathyroid glands is the most frequent complication after total thyroidectomy.

OBJECTIVE

To determine the effectiveness of near-infrared autofluorescence (NIRAF) in reducing postoperative hypocalcemia following total thyroidectomy.

METHODS

PubMed, Scopus, and Google Scholar databases were searched. Randomised trials reporting at least one hypocalcemia outcome following total thyroidectomy using NIRAF were included.

RESULTS

The qualitative data synthesis comprised 1363 patients from nine randomised studies, NIRAF arm = 636 cases and non-NIRAF arm = 637 cases. There was a statistically significant difference in the overall rate of hypocalcemia log(OR) = -0.7 [(-1.01, -0.40), M-H, REM, CI = 95%] and temporary hypocalcemia log(OR) = -0.8 [(-1.01, -0.59), M-H, REM, CI = 95%] favouring the NIRAF. The difference in the rate of permanent hypocalcemia log(OR) = -1.09 [(-2.34, 0.17), M-H, REM, CI = 95%] between the two arms was lower in the NIRAF arm but was not statistically significant.

CONCLUSIONS

NIRAF during total thyroidectomy helps in reducing postoperative hypocalcemia. Level of evidence-1.

Interpretation and use of a decision support tool for multiple treatment options: a combined randomised controlled trial and survey of medical students

OBJECTIVES

To investigate medical students' ability to interpret evidence, as well as their self-assessed understandability, perceived usefulness and preferences for design alternatives in an interactive decision support tool, displaying GRADE evidence summaries for multiple treatment options (Making Alternative Treatment CHoices Intuitive and Trustworthy, MATCH-IT).

DESIGN

A combined randomised controlled trial and survey. Participants were presented with a clinical scenario and randomised to one of two versions of the MATCH-IT tool (A/B), instructed to explore the evidence and decide on a recommendation. Participants answered a questionnaire assessing interpretation, treatment recommendation self-assessed understandability and perceived usefulness before exposure to the other MATCH-IT version and asked questions on design preferences.

SETTING

Online lecture in an evidence-based medicine (EBM) introductory course.

PARTICIPANTS

149 third-year medical students. 52% (n=77) had 6 months of clinical training and 48% (n=72) had preclinical training only.

INTERVENTIONS

The MATCH-IT tool version A uses colour coding to categorise interventions by magnitude and direction of effects and displays all outcomes in a table on entry. Version B has no colour coding, and the user must decide which outcomes to display in the table.

MAIN OUTCOME MEASURES

Interpretation of evidence, treatment recommendation, perceived usefulness and understandability, preference for format and design alternatives.

RESULTS

82.5% (n=123) of medical students correctly answered ≥4 out of 5 multiple choice questions assessing interpretation of data. 75.8% (n=114) of students made a treatment recommendation in accordance with an expert panel for the same clinical scenario. 87.2% (n=130) found the tool understandable while 91.9% perceived the tool as useful in addressing the clinical scenario.

CONCLUSION

Medical students with no prior training in EBM can interpret and use the MATCH-IT tool. Certain design alternatives were preferred but had no bearing on interpretation of evidence or understandability of the tool.

Comparison of ointment-based agents after excisional procedures for hemorrhoidal disease: a network meta-analysis of randomized controlled trials

INTRODUCTION

Efficient postoperative pain control is important after hemorrhoidal surgery. Although several locally applied medications have been used, current evidence regarding the optimal strategy is still conflicting. This network meta-analysis assessed analgesic efficacy and safety of the various topical medications in patients submitted to excisional procedures for hemorrhoids.

METHODS

The present study followed the Cochrane Handbook for Systematic Reviews of Interventions and the PRISMA guidelines. The last systematic literature screening was performed at 15 June 2023. Comparisons were based on a random effects multivariate network meta-analysis under a Bayesian framework.

RESULTS

Overall, 26 RCTs and 2132 patients were included. Regarding postoperative pain, EMLA cream (surface under the cumulative ranking curve (SUCRA) 80.3%) had the highest ranking at 12-h endpoint, while aloe vera cream (SUCRA 82.36%) scored first at 24 h. Metronidazole ointments had the highest scores at 7 and 14 days postoperatively. Aloe vera had the best analgesic profile (24-h SUCRA 84.8% and 48-h SUCRA 80.6%) during defecation. Lidocaine (SUCRA 87.9%) displayed the best performance regarding overall morbidity rates.

CONCLUSIONS

Due to the inconclusive results and several study limitations, further RCTs are required.

Data visualisation approaches for component network meta-analysis: visualising the data structure

BACKGROUND

Health and social care interventions are often complex and can be decomposed into multiple components. Multicomponent interventions are often evaluated in randomised controlled trials. Across trials, interventions often have components in common which are given alongside other components which differ across trials. Multicomponent interventions can be synthesised using component NMA (CNMA). CNMA is limited by the structure of the available evidence, but it is not always straightforward to visualise such complex evidence networks. The aim of this paper is to develop tools to visualise the structure of complex evidence networks to support CNMA.

METHODS

We performed a citation review of two key CNMA methods papers to identify existing published CNMA analyses and reviewed how they graphically represent intervention complexity and comparisons across trials. Building on identified shortcomings of existing visualisation approaches, we propose three approaches to standardise visualising the data structure and/or availability of data: CNMA-UpSet plot, CNMA heat map, CNMA-circle plot. We use a motivating example to illustrate these plots.

RESULTS

We identified 34 articles reporting CNMAs. A network diagram was the most common plot type used to visualise the data structure for CNMA (26/34 papers), but was unable to express the complex data structures and large number of components and potential combinations of components associated with CNMA. Therefore, we focused visualisation development around representing the data structure of a CNMA more completely. The CNMA-UpSet plot presents arm-level data and is suitable for networks with large numbers of components or combinations of components. Heat maps can be utilised to inform decisions about which pairwise interactions to consider for inclusion in a CNMA model. The CNMA-circle plot visualises the combinations of components which differ between trial arms and offers flexibility in presenting additional information such as the number of patients experiencing the outcome of interest in each arm.

CONCLUSIONS

As CNMA becomes more widely used for the evaluation of multicomponent interventions, the novel CNMA-specific visualisations presented in this paper, which improve on the limitations of existing visualisations, will be important to aid understanding of the complex data structure and facilitate interpretation of the CNMA results.

The fraternal birth-order effect as a statistical artefact: convergent evidence from probability calculus, simulated data, and multiverse meta-analysis

The fraternal-birth order effect (FBOE) is a research claim which states that each older brother increases the odds of homosexual orientation in men via an immunoreactivity process known as the maternal immune hypothesis. Importantly, older sisters supposedly either do not affect these odds, or affect them to a lesser extent. Consequently, the fraternal birth-order effect predicts that the association between the number of older brothers and homosexual orientation in men is greater in magnitude than any association between the number of older sisters and homosexual orientation. This difference in magnitude represents the main theoretical estimand of the FBOE. In addition, no comparable effects should be observable among homosexual vs heterosexual women. Here, we triangulate the empirical foundations of the FBOE from three distinct, informative perspectives, complementing each other: first, drawing on basic probability calculus, we deduce mathematically that the body of statistical evidence used to make inferences about the main theoretical estimand of the FBOE rests on incorrect statistical reasoning. In particular, we show that throughout the literature researchers ascribe to the false assumptions that effects of family size should be adjusted for and that this could be achieved through the use of ratio variables. Second, using a data-simulation approach, we demonstrate that by using currently recommended statistical practices, researchers are bound to frequently draw incorrect conclusions. And third, we re-examine the empirical evidence of the fraternal birth-order effect in men and women by using a novel specification-curve and multiverse approach to meta-analysis (64 male and 17 female samples, N = 2,778,998). When analyzed correctly, the specific association between the number of older brothers and homosexual orientation is small, heterogenous in magnitude, and apparently not specific to men. In addition, existing research evidence seems to be exaggerated by small-study effects.

Data visualisation in scoping reviews and evidence maps on health topics: a cross-sectional analysis

BACKGROUND

Scoping reviews and evidence maps are forms of evidence synthesis that aim to map the available literature on a topic and are well-suited to visual presentation of results. A range of data visualisation methods and interactive data visualisation tools exist that may make scoping reviews more useful to knowledge users. The aim of this study was to explore the use of data visualisation in a sample of recent scoping reviews and evidence maps on health topics, with a particular focus on interactive data visualisation.

METHODS

Ovid MEDLINE ALL was searched for recent scoping reviews and evidence maps (June 2020-May 2021), and a sample of 300 papers that met basic selection criteria was taken. Data were extracted on the aim of each review and the use of data visualisation, including types of data visualisation used, variables presented and the use of interactivity. Descriptive data analysis was undertaken of the 238 reviews that aimed to map evidence.

RESULTS

Of the 238 scoping reviews or evidence maps in our analysis, around one-third (37.8%) included some form of data visualisation. Thirty-five different types of data visualisation were used across this sample, although most data visualisations identified were simple bar charts (standard, stacked or multi-set), pie charts or cross-tabulations (60.8%). Most data visualisations presented a single variable (64.4%) or two variables (26.1%). Almost a third of the reviews that used data visualisation did not use any colour (28.9%). Only two reviews presented interactive data visualisation, and few reported the software used to create visualisations.

CONCLUSIONS

Data visualisation is currently underused by scoping review authors. In particular, there is potential for much greater use of more innovative forms of data visualisation and interactive data visualisation. Where more innovative data visualisation is used, scoping reviews have made use of a wide range of different methods. Increased use of these more engaging visualisations may make scoping reviews more useful for a range of stakeholders.

Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences

Meta-analysis is a quantitative way of synthesizing results from multiple studies to obtain reliable evidence of an intervention or phenomenon. Indeed, an increasing number of meta-analyses are conducted in environmental sciences, and resulting meta-analytic evidence is often used in environmental policies and decision-making. We conducted a survey of recent meta-analyses in environmental sciences and found poor standards of current meta-analytic practice and reporting. For example, only ~ 40% of the 73 reviewed meta-analyses reported heterogeneity (variation among effect sizes beyond sampling error), and publication bias was assessed in fewer than half. Furthermore, although almost all the meta-analyses had multiple effect sizes originating from the same studies, non-independence among effect sizes was considered in only half of the meta-analyses. To improve the implementation of meta-analysis in environmental sciences, we here outline practical guidance for conducting a meta-analysis in environmental sciences. We describe the key concepts of effect size and meta-analysis and detail procedures for fitting multilevel meta-analysis and meta-regression models and performing associated publication bias tests. We demonstrate a clear need for environmental scientists to embrace multilevel meta-analytic models, which explicitly model dependence among effect sizes, rather than the commonly used random-effects models. Further, we discuss how reporting and visual presentations of meta-analytic results can be much improved by following reporting guidelines such as PRISMA-EcoEvo (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Ecology and Evolutionary Biology). This paper, along with the accompanying online tutorial, serves as a practical guide on conducting a complete set of meta-analytic procedures (i.e., meta-analysis, heterogeneity quantification, meta-regression, publication bias tests and sensitivity analysis) and also as a gateway to more advanced, yet appropriate, methods.

Vitruvian plot: a visualisation tool for multiple outcomes in network meta-analysis

OBJECTIVE

A network meta-analysis (NMA) usually assesses multiple outcomes across several treatment comparisons. The Vitruvian plot aims to facilitate communication of multiple outcomes from NMAs to patients and clinicians.

METHODS

We developed this tool following the recommendations on the communication of benefit-risk information from the available literature. We collected and implemented feedback from researchers, statisticians, methodologists, clinicians and people with lived experience of physical and mental health issues.

RESULTS

We present the Vitruvian plot, which graphically presents absolute estimates and relative performance of competing interventions against a common comparator for several outcomes of interest. We use two alternative colour schemes to highlight either the strength of statistical evidence or the confidence in the evidence. Confidence in the evidence is evaluated across six domains (within-study bias, reporting bias, indirectness, imprecision, heterogeneity and incoherence) using the Confidence in Network Meta-Analysis (CINeMA) system.

CONCLUSIONS

The Vitruvian plot allows reporting of multiple outcomes from NMAs, with colourings appropriate to inform credibility of the presented evidence.

Use of mixed-type data clustering algorithm for characterizing temporal and spatial distribution of biosecurity border detections of terrestrial non-indigenous species

Appropriate inspection protocols and mitigation strategies are a critical component of effective biosecurity measures, enabling implementation of sound management decisions. Statistical models to analyze biosecurity surveillance data are integral to this decision-making process. Our research focuses on analyzing border interception biosecurity data collected from a Class A Nature Reserve, Barrow Island, in Western Australia and the associated covariates describing both spatial and temporal interception patterns. A clustering analysis approach was adopted using a generalization of the popular k-means algorithm appropriate for mixed-type data. The analysis approach compared the efficiency of clustering using only the numerical data, then subsequently including covariates to the clustering. Based on numerical data only, three clusters gave an acceptable fit and provided information about the underlying data characteristics. Incorporation of covariates into the model suggested four distinct clusters dominated by physical location and type of detection. Clustering increases interpretability of complex models and is useful in data mining to highlight patterns to describe underlying processes in biosecurity and other research areas. Availability of more relevant data would greatly improve the model. Based on outcomes from our research we recommend broader use of cluster models in biosecurity data, with testing of these models on more datasets to validate the model choice and identify important explanatory variables.

Applying generalized funnel plots to help design statistical analyses

Researchers across many fields routinely analyze trial data using Null Hypothesis Significance Tests with zero null and p < 0.05. To promote thoughtful statistical testing, we propose a visualization tool that highlights practically meaningful effects when calculating sample sizes. The tool re-purposes and adapts funnel plots, originally developed for meta-analyses, after generalizing them to cater for meaningful effects. As with traditional sample size calculators, researchers must nominate anticipated effect sizes and variability alongside the desired power. The advantage of our tool is that it simultaneously presents sample sizes needed to adequately power tests for equivalence, for non-inferiority and for superiority, each considered at up to three alpha levels and in positive and negative directions. The tool thus encourages researchers at the design stage to think about the type and level of test in terms of their research goals, costs of errors, meaningful effect sizes and feasible sample sizes. An R-implementation of the tool is available on-line.

PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews

The methods and results of systematic reviews should be reported in sufficient detail to allow users to assess the trustworthiness and applicability of the review findings. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement was developed to facilitate transparent and complete reporting of systematic reviews and has been updated (to PRISMA 2020) to reflect recent advances in systematic review methodology and terminology. Here, we present the explanation and elaboration paper for PRISMA 2020, where we explain why reporting of each item is recommended, present bullet points that detail the reporting recommendations, and present examples from published reviews. We hope that changes to the content and structure of PRISMA 2020 will facilitate uptake of the guideline and lead to more transparent, complete, and accurate reporting of systematic reviews.

Meta-analysis as a system of springs

Meta-analysis results are usually presented in forest plots, which show the individual study results and the summary effect along with their confidence intervals. In this paper, we propose a system of linear springs as a mechanical analogue of meta-analysis that enables visualization and enhances intuition. The length of a spring corresponds to a study treatment effect and the stiffness of the spring corresponds to its inverse variance. To synthesize study springs we use two main operations: connection in parallel and connection in series. We show the equivalence between meta-analysis and linear springs for fixed effect and random effects pairwise meta-analysis and we also derive indirect treatment effects. We use examples to illustrate the different meta-analytical schemes using the corresponding system of springs. The proposed visualization can serve as an educational tool, especially useful for researchers with no statistical background. The analogy between meta-analysis and springs facilitates intuition for notions such as heterogeneity and the differences between fixed and random effects meta-analysis.

The orchard plot: Cultivating a forest plot for use in ecology, evolution, and beyond

"Classic" forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution, meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a "forest-like plot," showing point estimates (with 95% confidence intervals [CIs]) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the "orchard plot." Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also include 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

Beyond the forest plot: The drapery plot

In the era of the "reproducibility crisis" and the "P-value controversy" new ways of presentation and interpretation of the results of a meta-analysis are desirable. One suggestion that has been made for single studies almost six decades ago and taken up now and then is the P-value function. For a given outcome, this function assigns a P-value to each possible hypothetical value, given the data. Moreover, the P-value function simultaneously provides two-sided confidence intervals for all possible alpha levels. An application to meta-analysis, while suggested early, has not been widely established. We introduce the drapery plot that presents the P-value function for all individual studies and pooled estimates in a meta-analysis as curves and the prediction range for a single future study. We also present a scaled variant with the test statistic on the y-axis. Both plots visualize the full information of a pairwise meta-analysis. We see a drapery plot as a complementary figure to a forest plot. It may be even an alternative in meta-analyses with many studies where forest plots tend to become very large and complex.