The effect of bias on the magnitude of clinical outcomes in periodontology: a pilot study

Application of the Composite Quality Score (CQS-2B) versus Cochrane's Risk of Bias tool (Version 2) in systematic reviews of clinical trials - an exploratory study

Objectives

To explore whether systematic review conclusions generated from Cochrane's second version of its Risk of Bias tool (RoB 2) for trial appraisal differ when the Composite Quality Score, Version 2.B (CQS-2B) is used instead and to develop a testable hypothesis based on these findings.

Methods

PubMed was searched for one single systematic review. From the review's accepted trials, data concerning effect estimates and overall bias risk according to the RoB 2 tool were extracted. All trial reports were appraised again using the CQS-2B. Datasets were stratified according to overall bias risk (RoB 2) or corroboration (C-) level (CQS-2B). The effect estimates from trials with 'low bias risk' (RoB 2) and with highest C-level (CQS-2B) were pooled separately. These pooled effect estimates were statistically and all clinical conclusions qualitatively compared.

Results

The pooled effect estimates for trials with 'low bias risk' (RoB 2) were -0.07, 95% CI: -0.10 to -0.04 (I2 = 0.0%) and for the highest C-levels (CQS-2B) 0.08, 95% CI: -0.12 to -0.04 (I2 = 57.0%). The difference was statistically not significant (p = 0.70). Contrary to the RoB 2 tool, no clinical conclusions in line with the CQS-2B were made, because the effect estimates were judged to be erroneously overestimated, due to high risk of bias.

Conclusion

A testable hypothesis was generated suggesting that trial appraisal using the CQS-2B may provide more conservative conclusions based on similar data than with the RoB 2 tool.

Allocation concealment appraisal of clinical therapy trials using the extended Composite Quality Score (CQS-2)-An empirically based update

Objectives

The objective of this study was to revise CQS-2/Criterion II concerning allocation concealment appraisal for prospective, controlled clinical therapy trials.

Methods

Meta-analyses of trials with inadequate allocation concealment were tested for in-between trial heterogeneity (I² > 0) due to imbalances in baseline variables. Meta-analyses with positive test results were used as a basis to deduce criteria for adequate allocation concealment. The CQS-2/Criterion II was reformulated in line with the findings.

Result

One suitable meta-analysis was identified. Two forest plots with data from five and four trials with inadequate/unclear allocation concealment were selected for testing. In addition, a total of five trials with adequate allocation concealment were identified. The meta-analysis test results were positive, and keywords for the judgment of adequate allocation concealment were extracted verbatim from the text of the meta-analysis. The extracted keywords indicated "central allocation" as the main criterion for adequate allocation concealment. Criterion II of the CQS-2 was revised accordingly.

Conclusion

Criterion II of the CQS-2 trial appraisal tool was revised. The revised appraisal tool was specified as version CQS-2B.

Extension of the Composite Quality Score (CQS) as an appraisal tool for prospective, controlled clinical therapy trials-A systematic review of meta-epidemiological evidence

AIM

To conduct a survey of current meta-epidemiological studies to identify additional trial design characteristics that may be associated with significant over- or underestimation of the treatment effect and to use such identified characteristics as a basis for the formulation of new CQS appraisal criteria.

MATERIALS AND METHODS

We retrieved eligible studies from two systematic reviews on this topic (latest search May 2015) and searched the databases PubMed and Embase for further studies from June 2015 -March 2022. All data were extracted by one author and verified by another. Sufficiently homogeneous estimates from single studies were pooled using random-effects meta-analysis. Trial design characteristics associated with statistically significant estimates from single datasets (which could not be pooled) and meta-analyses were used as a basis to formulate new or amend existing CQS criteria.

RESULTS

A total of 38 meta-epidemiological studies were identified. From these, seven trial design characteristics associated with statistically significant over- or underestimation of the true therapeutic effect were found.

CONCLUSION

One new criterion concerning double-blinding was added to the CQS, and the original criteria for concealing the random allocation sequence and for minimum sample size were amended.

Influence of Sponsorship Bias on Treatment Effect Size Estimates in Randomized Trials of Oral Health Interventions: A Meta-epidemiological Study

BACKGROUND

In this meta-epidemiological study, we aimed to examine associations between treatment effect size estimates and sponsorship bias in oral health randomized clinical trials.

METHODS

We selected oral health related meta-analyses that included a minimum of five randomized controlled trials. We extracted data, in duplicate, related to influence of sponsorship bias. We quantified the extent of bias associated with influence of sponsorship on the magnitude of effect size estimates of continuous variables using a two-level meta-meta-analytic approach with random-effects models to allow for intra- and inter-meta-analysis heterogeneity.

RESULTS

We initially identified 540 randomized trials included in 64 meta-analyses. Risk of sponsorship bias was judged as being "unclear" in 72.8% (n = 393) of the trials, while it was assessed as "low" in 16.7% (n = 90) and as "high" in 10.6% (n = 57) of the trials. Using a meta-epidemiological analysis (37 meta-analyses, including 328 trials that analyzed 85,934 patients), we identified statistically significant larger treatment effect size estimates in trials that had "high or unclear" risk of sponsorship bias (difference in treatment effect size estimates=0.10; 95% confidence intervals: 0.02 to 0.19) than in trials that had "low" risk of sponsorship bias.

CONCLUSIONS

We identified significant differences in treatment effect size estimates between dental trials based on sponsorship bias. Treatment effect size estimates were 0.10 larger in trials with "high or unclear" risk of sponsorship bias.

PRACTICAL IMPLICATIONS

Clinicians should have an adequate knowledge of sponsorship bias in a clinical trial and be able to estimate the degree to which the conclusions of a systematic review are synthesized and interpreted, based on trials with low risk of sponsorship bias.

Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions

Background

Recent methodologic evidence suggests that lack of blinding in randomized trials can result in under- or overestimation of the treatment effect size. The objective of this study is to quantify the extent of bias associated with blinding in randomized controlled trials of oral health interventions.

Methods

We selected all oral health meta-analyses that included a minimum of five randomized controlled trials. We extracted data, in duplicate, related to nine blinding-related criteria, namely: patient blinding, assessor blinding, care-provider blinding, investigator blinding, statistician blinding, blinding of both patients and assessors, study described as “double blind”, blinding of patients, assessors, and care providers concurrently, and the appropriateness of blinding. We quantified the impact of bias associated with blinding on the magnitude of effect size using a two-level meta-meta-analytic approach with a random effects model to allow for intra- and inter-meta-analysis heterogeneity.

Results

We identified 540 randomized controlled trials, included in 64 meta-analyses, analyzing data from 137,957 patients. We identified significantly larger treatment effect size estimates in trials that had inadequate patient blinding (difference in treatment effect size = 0.12; 95% CI: 0.00 to 0.23), lack of blinding of both patients and assessors (difference = 0.19; 95% CI: 0.06 to 0.32), and lack of blinding of patients, assessors, and care-providers concurrently (difference = 0.14; 95% CI: 0.03 to 0.25). In contrast, assessor blinding (difference = 0.06; 95% CI: -0.06 to 0.18), caregiver blinding (difference = 0.02; 95% CI: -0.04 to 0.09), principal-investigator blinding (difference = − 0.02; 95% CI: -0.10 to 0.06), describing a trial as “double-blind” (difference = 0.09; 95% CI: -0.05 to 0.22), and lack of an appropriate method of blinding (difference = 0.06; 95% CI: -0.06 to 0.18) were not associated with over- or underestimated treatment effect size.

Conclusions

We found significant differences in treatment effect size estimates between oral health trials based on lack of patient and assessor blinding. Treatment effect size estimates were 0.19 and 0.14 larger in trials with lack of blinding of both patients and assessors and blinding of patients, assessors, and care-providers concurrently. No significant differences were identified in other blinding criteria. Investigators of oral health systematic reviews should perform sensitivity analyses based on the adequacy of blinding in included trials.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0491-0) contains supplementary material, which is available to authorized users.

Impact of Selection Bias on Treatment Effect Size Estimates in Randomized Trials of Oral Health Interventions: A Meta-epidemiological Study

Emerging evidence suggests that design flaws of randomized controlled trials can result in over- or underestimation of the treatment effect size (ES). The objective of this study was to examine associations between treatment ES estimates and adequacy of sequence generation, allocation concealment, and baseline comparability among a sample of oral health randomized controlled trials. For our analysis, we selected all meta-analyses that included a minimum of 5 oral health randomized controlled trials and used continuous outcomes. We extracted data, in duplicate, related to items of selection bias (sequence generation, allocation concealment, and baseline comparability) in the Cochrane Risk of Bias tool. Using a 2-level meta-meta-analytic approach with a random effects model to allow for intra- and inter-meta-analysis heterogeneity, we quantified the impact of selection bias on the magnitude of ES estimates. We identified 64 meta-analyses, including 540 randomized controlled trials analyzing 137,957 patients. Sequence generation was judged to be adequate (at low risk of bias) in 32% ( n = 173) of trials, and baseline comparability was judged to be adequate in 77.8% of trials. Allocation concealment was unclear in the majority of trials ( n = 458, 84.8%). We identified significantly larger treatment ES estimates in trials that had inadequate/unknown sequence generation (difference in ES = 0.13; 95% CI: 0.01 to 0.25) and inadequate/unknown allocation concealment (difference in ES = 0.15; 95% CI: 0.02 to 0.27). In contrast, baseline imbalance (difference in ES = 0.01, 95% CI: –0.09 to 0.12) was not associated with inflated or underestimated ES. In conclusion, treatment ES estimates were 0.13 and 0.15 larger in trials with inadequate/unknown sequence generation and inadequate/unknown allocation concealment, respectively. Therefore, authors of systematic reviews using oral health randomized controlled trials should perform sensitivity analyses based on the adequacy of sequence generation and allocation concealment.

Positive association between conflicts of interest and reporting of positive results in randomized clinical trials in dentistry

BACKGROUND

The relationship between industry funding and study results has been explored widely in medicine but not in dentistry. The authors aimed to assess the relationship between conflicts of interest (COIs) and study results.

METHODS

The authors assessed all randomized clinical trials (RCTs) published between July 2010 and June 2012 in the 10 dental journals with the highest impact factors in dentistry. The authors used three definitions of COI and explored their associations with positive study results.

RESULTS

Depending on the definition of COI, the odds ratio for reporting positive results varied between 2.40 (95 percent confidence interval [CI], 1.16-5.13) and 9.19 (95 percent CI, 1.71-170.64). The authors found no association between positive study results and journal of publication or area of practice.

CONCLUSIONS

RCTs in which authors have some type of COI are more likely to have results that support the intervention being assessed.

PRACTICAL IMPLICATIONS

When reviewing the results of RCTs, clinicians need to be aware of the association between reporting positive study results and the type of COI disclosure and be even more careful when critically appraising and applying their results.

Risk of bias and magnitude of effect in orthodontic randomized controlled trials: a meta-epidemiological review

AIM

To assess the risk of bias (RoB) in a subset of randomized controlled trials (RCTs) published in orthodontic journals using the Cochrane RoB tool and to identify associations between domain RoB assessment and treatment effect estimates.

MATERIALS AND METHODS

Fifty consecutive issues of four major orthodontic journals were electronically searched to identify RCTs. The quality of the included studies was assessed using the Cochrane RoB tool, which involves seven domains rated as 'low', 'unclear' or 'high': random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting, and other threats to internal validity. Estimates and confidence intervals (CIs) were recorded or calculated where possible for binary and continuous outcome measures. Meta-regression models were employed to assess the impact of RoB per domain on the magnitude of treatment effect.

RESULTS

One hundred and one eligible studies involving 128 pair-wise comparisons were retrieved. Blinding of outcome assessors and incomplete outcome data were frequently judged as 'high' for RoB both for studies with binary and continuous outcome (42.9 and 48.8 per cent, respectively). For binary outcomes, high RoB regarding random sequence generation [odds ratio (OR): 5.97, 95% CI: 2.03, 17.63, P-value: 0.002] and incomplete outcome data (OR: 4.07, 95% CI: 1.03, 16.15, P-value: 0.05) were more likely to provide exaggerated effect estimates.

CONCLUSIONS

There is a need for improved clinical trial methodology and reporting, in order to avoid inflated associations and erroneous conclusions.

Methodological characteristics and treatment effect sizes in oral health randomised controlled trials: Is there a relationship? Protocol for a meta-epidemiological study

INTRODUCTION

It is fundamental that randomised controlled trials (RCTs) are properly conducted in order to reach well-supported conclusions. However, there is emerging evidence that RCTs are subject to biases which can overestimate or underestimate the true treatment effect, due to flaws in the study design characteristics of such trials. The extent to which this holds true in oral health RCTs, which have some unique design characteristics compared to RCTs in other health fields, is unclear. As such, we aim to examine the empirical evidence quantifying the extent of bias associated with methodological and non-methodological characteristics in oral health RCTs.

METHODS AND ANALYSIS

We plan to perform a meta-epidemiological study, where a sample size of 60 meta-analyses (MAs) including approximately 600 RCTs will be selected. The MAs will be randomly obtained from the Oral Health Database of Systematic Reviews using a random number table; and will be considered for inclusion if they include a minimum of five RCTs, and examine a therapeutic intervention related to one of the recognised dental specialties. RCTs identified in selected MAs will be subsequently included if their study design includes a comparison between an intervention group and a placebo group or another intervention group. Data will be extracted from selected trials included in MAs based on a number of methodological and non-methodological characteristics. Moreover, the risk of bias will be assessed using the Cochrane Risk of Bias tool. Effect size estimates and measures of variability for the main outcome will be extracted from each RCT included in selected MAs, and a two-level analysis will be conducted using a meta-meta-analytic approach with a random effects model to allow for intra-MA and inter-MA heterogeneity.

ETHICS AND DISSEMINATION

The intended audiences of the findings will include dental clinicians, oral health researchers, policymakers and graduate students. The aforementioned will be introduced to the findings through workshops, seminars, round table discussions and targeted individual meetings. Other opportunities for knowledge transfer will be pursued such as key dental conferences. Finally, the results will be published as a scientific report in a dental peer-reviewed journal.

Alveolar ridge preservation. A systematic review

OBJECTIVE

The objective of this paper is to examine the effect of alveolar ridge preservation (ARP) compared to unassisted socket healing.

METHODS

Systematic review with electronic and hand search was performed. Randomised controlled trials (RCT), controlled clinical trials (CCT) and prospective cohort studies were eligible.

RESULTS

Eight RCTs and six CCTs were identified. Clinical heterogeneity did not allow for meta-analysis. Average change in clinical alveolar ridge (AR) width varied between -1.0 and -3.5 ± 2.7 mm in ARP groups and between -2.5 and -4.6 ± 0.3 mm in the controls, resulting in statistically significantly smaller reduction in the ARP groups in five out of seven studies. Mean change in clinical AR height varied between +1.3 ± 2.0 and -0.7 ± 1.4 mm in the ARP groups and between -0.8 ± 1.6 and -3.6 ± 1.5 mm in the controls. Height reduction in the ARP groups was statistically significantly less in six out of eight studies. Histological analysis indicated various degrees of new bone formation in both groups. Some graft interfered with the healing. Two out of eight studies reported statistically significantly more trabecular bone formation in the ARP group. No superiority of one technique for ARP could be identified; however, in certain cases guided bone regeneration was most effective. Statistically, significantly less augmentation at implant placement was needed in the ARP group in three out of four studies. The strength of evidence was moderate to low.

CONCLUSIONS

Post-extraction resorption of the AR might be limited, but cannot be eliminated by ARP, which at histological level does not always promote new bone formation. RCTs with unassisted socket healing and implant placement in the ARP studies are needed to support clinical decision making.

CLINICAL RELEVANCE

This systematic review reports not only on the clinical and radiographic outcomes, but also evaluates the histological appearance of the socket, along with site specific factors, patient-reported outcomes, feasibility of implant placement and strength of evidence, which will facilitate the decision making process in the clinical practice.

Clinical outcomes of single-visit oral prophylaxis: a practice-based randomised controlled trial

BACKGROUND

Practice-based general dental practitioners routinely provide "scale and polish" or "oral prophylaxis" to patients attending their practices. Despite its routine provision, there is no evidence to support the clinical effectiveness of single-visit scale and polish, nor the frequency at which it should be provided. A recent systematic review recommended that future trials investigating scale and polish should involve dental practice patients.

METHODS

A practice-based parallel randomised controlled trial with 24-month follow-up was conducted. Healthy adults (Basic Periodontal Examination [BPE] codes <3) were randomly assigned to 3 groups (6-month, 12-month, or 24-month interval between scale and polish). The primary outcome was gingival bleeding with the hypothesis that 6-monthly scale and polish would result in lower prevalence than 12-month or 24-month frequency. Follow-up measurements were recorded by examiners blinded to the allocation. 125, 122 and 122 participants were randomised to the 6-month, 12-month and 24-month groups respectively. Complete data set analyses were conducted for 307 participants: 107, 100, and 100 in the 6-month, 12-month and 24-month groups respectively. Chi-square test and ANOVA were used to compare treatment groups at follow-up. Logistic regression and ANCOVA were used to estimate the relationship between outcome and treatment group, adjusted for baseline values. Multiple imputation analyses were also carried out for participants with incomplete data sets.

RESULTS

Prevalence of gingival bleeding at follow-up was 78.5% (6-month), 78% (12-month) and 82% (24-month) (p = 0.746). There were no statistically significant differences between groups with respect to follow-up prevalence of plaque and calculus. Statistically significant differences detected in the amount (millimetres) of calculus were too small to be clinically significant. Seventeen (4.6%) participants were withdrawn from the trial to receive additional treatment.

CONCLUSIONS

This trial could not identify any differences in outcomes for single-visit scale and polish provided at 6, 12 and 24 month frequencies for healthy patients (with no significant periodontal disease). However, this is the first trial of scale and polish which has been conducted in a general practice setting and the results are not conclusive. Larger trials with more comprehensive measurement and long-term follow up need to be undertaken to provide a firm evidence base for this intervention. This trial informs the design of future practice-based trials on this subject.

Randomisation to protect against selection bias in healthcare trials

BACKGROUND

Randomised trials use the play of chance to assign participants to comparison groups. The unpredictability of the process, if not subverted, should prevent systematic differences between comparison groups (selection bias). Differences due to chance will still occur and these are minimised by randomising a sufficiently large number of people.

OBJECTIVES

To assess the effects of randomisation and concealment of allocation on the results of healthcare studies.

SEARCH STRATEGY

We searched the Cochrane Methodology Register, MEDLINE, SciSearch and reference lists up to September 2009. In addition, we screened articles citing included studies (ISI Science Citation Index) and papers related to included studies (PubMed).

SELECTION CRITERIA

Eligible study designs were cohorts of studies, systematic reviews or meta-analyses of healthcare interventions that compared random allocation versus non-random allocation or adequate versus inadequate/unclear concealment of allocation in randomised trials. Outcomes of interest were the magnitude and direction of estimates of effect and imbalances in prognostic factors.

DATA COLLECTION AND ANALYSIS

We retrieved and assessed studies that appeared to meet the inclusion criteria independently. At least two review authors independently appraised methodological quality and extracted information. We prepared tabular summaries of the results for each comparison and assessed the results across studies qualitatively to identify common trends or discrepancies.

MAIN RESULTS

A total of 18 studies (systematic reviews or meta-analyses) met our inclusion criteria. Ten compared random allocation versus non-random allocation and nine compared adequate versus inadequate or unclear concealment of allocation within controlled trials. All studies were at high risk of bias.For the comparison of randomised versus non-randomised studies, four comparisons yielded inconclusive results (differed between outcomes or different modes of analysis); three comparisons showed similar results for random and non-random allocation; two comparisons had larger estimates of effect in non-randomised studies than in randomised trials; and two comparisons had larger estimates of effect in randomised than in non-randomised studies.Five studies found larger estimates of effect in trials with inadequate concealment of allocation than in trials with adequate concealment. The four other studies did not find statistically significant differences.

AUTHORS' CONCLUSIONS

The results of randomised and non-randomised studies sometimes differed. In some instances non-randomised studies yielded larger estimates of effect and in other instances randomised trials yielded larger estimates of effect. The results of controlled trials with adequate and inadequate/unclear concealment of allocation sometimes differed. When differences occurred, most often trials with inadequate or unclear allocation concealment yielded larger estimates of effects relative to controlled trials with adequate allocation concealment. However, it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects from studies with non-random allocation or controlled trials with inadequate or unclear allocation concealment.

The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study

OBJECTIVE

To evaluate the association of adequate allocation concealment and patient blinding with estimates of treatment benefits in osteoarthritis trials.

METHODS

We performed a meta-epidemiologic study of 16 meta-analyses with 175 trials that compared therapeutic interventions with placebo or nonintervention control in patients with hip or knee osteoarthritis. We calculated effect sizes from the differences in means of pain intensity between groups at the end of followup divided by the pooled SD and compared effect sizes between trials with and trials without adequate methodology.

RESULTS

Effect sizes tended to be less beneficial in 46 trials with adequate allocation concealment compared with 112 trials with inadequate or unclear concealment of allocation (difference -0.15; 95% confidence interval [95% CI] -0.31, 0.02). Selection bias associated with inadequate or unclear concealment of allocation was most pronounced in meta-analyses with large estimated treatment benefits (P for interaction < 0.001), meta-analyses with high between-trial heterogeneity (P = 0.009), and meta-analyses of complementary medicine (P = 0.019). Effect sizes tended to be less beneficial in 64 trials with adequate blinding of patients compared with 58 trials without (difference -0.15; 95% CI -0.39, 0.09), but differences were less consistent and disappeared after accounting for allocation concealment. Detection bias associated with a lack of adequate patient blinding was most pronounced for nonpharmacologic interventions (P for interaction < 0.001).

CONCLUSION

Results of osteoarthritis trials may be affected by selection and detection bias. Adequate concealment of allocation and attempts to blind patients will minimize these biases.