Design and quality considerations for randomized controlled trials in systemic sclerosis

Chinese Neurosurgical Randomized Controlled Trials: Dynamics in Trial Implementation and Completion

BACKGROUND AND OBJECTIVES

The focus on evidence-based neurosurgery has led to a considerable amount of neurosurgical evidence based on randomized controlled trials (RCTs) being published. Nevertheless, there has been no systematic appraisal of China's contribution to RCTs. Information about the changes in characteristics of Chinese neurosurgical RCTs before and during the COVID-19 pandemic is limited. This study aims to perform a detailed examination and comprehensive analysis of the characteristics of Chinese neurosurgical RCTs and to examine the differences before and during the COVID-19 pandemic.

METHODS

We conducted a comprehensive database search including PubMed, Web of Science, Embase, and Cochrane Library up to March 2023, with a criterion of inclusion based on an impact factor above 0. We subsequently examined the design and quality parameters of the included RCTs and assessed the differences before and during the COVID-19 pandemic (based on follow-up ending before or after January 2020). Moreover, we investigated potential factors that may affect the quality and developmental trends of neurosurgical RCTs in China.

RESULTS

The main focus of the 91 neurosurgical RCTs was vascular disease (47.3%) and trauma (18.7%). Over half of the trials used Consolidated Standards of Reporting Trial diagrams (69.2%), and the majority compared nonsurgical treatments (63.7%). Larger trials tended to have better quality scores, but those with significant efficacy were less likely to have power calculations. Over time, there was an increase in the use of Consolidated Standards of Reporting Trial diagrams and well-specified outcomes. The COVID-19 pandemic may have hindered the completion of neurosurgical RCTs in China, but it has had little impact on the design and quality so far.

CONCLUSION

Chinese neurosurgeons have made significant progress in advancing neurosurgical RCTs despite challenges. However, shortcomings in sample size and power calculation need attention. Improving the rigor, rationality, and completeness of neurosurgical RCT design is crucial.

Neurosurgical Randomized Controlled Trials-Distance Travelled

BACKGROUND

The evidence base for many neurosurgical procedures has been limited. We performed a comprehensive and systematic analysis of study design, quality of reporting, and trial results of neurosurgical randomized controlled trials (RCTs).

OBJECTIVE

To systematically assess the design and quality characteristics of neurosurgical RCTs.

METHODS

From January 1961 to June 2016, RCTs with >5 patients assessing any 1 neurosurgical procedure against another procedure, nonsurgical treatment, or no treatment were retrieved from MEDLINE, Scopus, and Cochrane Library.

RESULTS

The median sample size in the 401 eligible RCTs was 73 patients with a mean patient age of 49.6. Only 111 trials (27.1%) described allocation concealment, 140 (34.6%) provided power calculations, and 117 (28.9%) were adequately powered. Significant efficacy or trend for efficacy was claimed in 226 reports (56.4%), no difference between the procedures was found in 166 trials (41.4%), and significant harm was reported in 9 trials (2.2%). Trials with a larger sample size were more likely to report randomization mode, specify allocation concealment, and power calculations (all P < .001). Government funding was associated with better specification of power calculations (P = .008) and of allocation concealment (P = .026), while industry funding was associated with reporting significant efficacy (P = .02). Reporting of funding, specification of randomization mode and primary outcomes, and mention of power calculations improved significantly (all, P < .05) over time.

CONCLUSION

Several aspects of the design and reporting of RCTs on neurosurgical procedures have improved over time. Better powered and accurately reported trials are needed in neurosurgery to deliver evidence-based care and achieve optimal outcomes.

Randomisation to protect against selection bias in healthcare trials

BACKGROUND

Randomised trials use the play of chance to assign participants to comparison groups. The unpredictability of the process, if not subverted, should prevent systematic differences between comparison groups (selection bias). Differences due to chance will still occur and these are minimised by randomising a sufficiently large number of people.

OBJECTIVES

To assess the effects of randomisation and concealment of allocation on the results of healthcare studies.

SEARCH STRATEGY

We searched the Cochrane Methodology Register, MEDLINE, SciSearch and reference lists up to September 2009. In addition, we screened articles citing included studies (ISI Science Citation Index) and papers related to included studies (PubMed).

SELECTION CRITERIA

Eligible study designs were cohorts of studies, systematic reviews or meta-analyses of healthcare interventions that compared random allocation versus non-random allocation or adequate versus inadequate/unclear concealment of allocation in randomised trials. Outcomes of interest were the magnitude and direction of estimates of effect and imbalances in prognostic factors.

DATA COLLECTION AND ANALYSIS

We retrieved and assessed studies that appeared to meet the inclusion criteria independently. At least two review authors independently appraised methodological quality and extracted information. We prepared tabular summaries of the results for each comparison and assessed the results across studies qualitatively to identify common trends or discrepancies.

MAIN RESULTS

A total of 18 studies (systematic reviews or meta-analyses) met our inclusion criteria. Ten compared random allocation versus non-random allocation and nine compared adequate versus inadequate or unclear concealment of allocation within controlled trials. All studies were at high risk of bias.For the comparison of randomised versus non-randomised studies, four comparisons yielded inconclusive results (differed between outcomes or different modes of analysis); three comparisons showed similar results for random and non-random allocation; two comparisons had larger estimates of effect in non-randomised studies than in randomised trials; and two comparisons had larger estimates of effect in randomised than in non-randomised studies.Five studies found larger estimates of effect in trials with inadequate concealment of allocation than in trials with adequate concealment. The four other studies did not find statistically significant differences.

AUTHORS' CONCLUSIONS

The results of randomised and non-randomised studies sometimes differed. In some instances non-randomised studies yielded larger estimates of effect and in other instances randomised trials yielded larger estimates of effect. The results of controlled trials with adequate and inadequate/unclear concealment of allocation sometimes differed. When differences occurred, most often trials with inadequate or unclear allocation concealment yielded larger estimates of effects relative to controlled trials with adequate allocation concealment. However, it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects from studies with non-random allocation or controlled trials with inadequate or unclear allocation concealment.

How to perform a critical analysis of a randomised controlled trial

Given the large amount of medical literature of varying methodological quality, care must be taken when translating the results of randomised controlled trials into clinical practice. To assist in this translation process, we provide a method that involves answering three main questions: 'Can I trust the results?' 'How do I understand the results?' and 'To whom do the results apply?' To answer the first question, we describe important items that help in judging the reliability of the findings. For the second question, we address the clinical and statistical significance of results by looking at the size and precision of the effect. Finally, we raise the issue of external validity and of reporting adverse effects to determine which patients may best benefit from the new intervention.

Randomized controlled trials in neurosurgery--how good are we?

BACKGROUND

The strongest evidence in medical clinical literature is represented by randomized controlled trials (RCTs). This study was designed to evaluate neurosurgically relevant RCTs published recently by neurosurgeons.

METHOD

A literature search in MEDLINE and EMBASE included all clinical studies published up to 30 June 2006. RCTs with neurosurgical relevance published by at least one author with affiliation to a neurosurgical department were selected. The number and characteristics of individual trials were recorded, and the quality of the trials with regard to study design, quality of reporting, and relevance for clinical practice was assessed by two different investigators using a modification of the Scottish Intercollegiate Guidelines Network methodology checklist. Changes of RCT quality over time as well as factors influencing the quality were analyzed.

FINDINGS

From the initial search results (MEDLINE n = 3,860, EMBASE n = 3,113 articles), 159 RCTs published by neurosurgeons were extracted for final evaluation. Of the RCTs, 62% have been published since 1995; 52% came from the USA, UK, and Germany. The median RCT sample size was 78 patients and the median follow-up 35.7 weeks. Fifty-two percent of all RCTs were of good, 37% of moderate, and 11% of bad quality, with an improvement over time. RCTs with financial funding and RCTs with a sample size of >78 patients were of significantly better quality. There were no major differences in the rating of the studies between the two investigators.

CONCLUSIONS

Only a fraction of neurosurgically relevant literature consists of RCTs, but the quality is satisfying and has significantly improved over the last years. An adequate sample size and sufficient financial support seem to be of substantial importance with regard to the quality of the study. Our data also show that by using a standardized checklist, the quality of trials can be reliably assessed by observers of different experience and educational levels.

A systematic review of trends in the methodological quality of randomized controlled trials in various research fields

OBJECTIVES

We sought to evaluate the trends in the methodological quality of randomized controlled trials in various medical fields.

STUDY DESIGN AND SETTING

Relevant studies were retrieved by the PubMed and the ISI Web of science databases.

RESULTS

Thirty-five out of 457 retrieved studies met the inclusion criteria. Twenty-one out of 35 selected studies reported significant improvement in at least one methodological quality factor. Overall quality scores were increased in 13 out of 26 studies providing relevant data. The most commonly separately examined key quality factors were allocation concealment and blinding in 13 out of 21 studies that reported relevant data. Allocation concealment was the quality characteristic most commonly reported as significantly improving during the reviewed period (in five out of eight studies reporting relevant comparative data).

CONCLUSION

Certain aspects of methodological quality have improved significantly over time, but others remain stagnant. Further efforts to improve study design, conduct, and reporting of randomized controlled trials are warranted.

Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials

BACKGROUND

Randomized trials without reported adequate allocation concealment have been shown to overestimate the benefit of experimental interventions. We investigated the robustness of conclusions drawn from meta-analyses to exclusion of such trials.

MATERIAL

Random sample of 38 reviews from The Cochrane Library 2003, issue 2 and 32 other reviews from PubMed accessed in 2002. Eligible reviews presented a binary effect estimate from a meta-analysis of randomized controlled trials as the first statistically significant result that supported a conclusion in favour of one of the interventions.

METHODS

We assessed the methods sections of the trials in each included meta-analysis for adequacy of allocation concealment. We replicated each meta-analysis using the authors' methods but included only trials that had adequate allocation concealment. Conclusions were defined as not supported if our result was not statistically significant.

RESULTS

Thirty-four of the 70 meta-analyses contained a mixture of trials with unclear or inadequate concealment as well as trials with adequate allocation concealment. Four meta-analyses only contained trials with adequate concealment, and 32, only trials with unclear or inadequate concealment. When only trials with adequate concealment were included, 48 of 70 conclusions (69%; 95% confidence interval: 56-79%) lost support. The loss of support mainly reflected loss of power (the total number of patients was reduced by 49%) but also a shift in the point estimate towards a less beneficial effect.

CONCLUSION

Two-thirds of conclusions in favour of one of the interventions were no longer supported if only trials with adequate allocation concealment were included.

Evidence from crossover trials: empirical evaluation and comparison against parallel arm trials

BACKGROUND

We aimed to evaluate empirically how crossover trial results are analysed in meta-analyses of randomized evidence and whether their results agree with parallel arm studies on the same questions.

METHODS

We used a systematic sample of Cochrane meta-analyses including crossover trials. We evaluated the methods of analysis for crossover results and compared the concordance of the estimated effect sizes in crossover vs parallel arm trials.

RESULTS

Of 334 screened reviews, 62 had crossover trials. Of those, 33 meta-analyses performed quantitative syntheses involving two-arm two-period crossover trials. There was large variability on how these trials were analysed; only one of the 33 meta-analyses stated that they used the data from both the first and second period with an appropriate paired approach. Nine meta-analyses used the first period data only and 14 gave no information at all on what they had done. Twenty-eight meta-analyses had both crossover (n = 137, sample size n = 7,162) and parallel arm (n = 132, sample size n = 11,398) trials. Effect sizes correlated well with the two types of designs (rho = 0.72). Differences on whether the summary effect had a P < 0.05 or not were common due to limited sample sizes. The summary relative odds ratio for parallel arm vs crossover designs for favourable outcomes was 0.87 (95% CI, 0.74-1.02).

CONCLUSIONS

Crossover designs may contribute evidence in a fifth of systematic reviews, but few meta-analyses make use of their full data. The results of crossover trials tend to agree with those of parallel arm trials, although there was a trend for more conservative treatment effect estimates in parallel arm trials.

Bias in clinical intervention research

Research on bias in clinical trials may help identify some of the reasons why investigators sometimes reach the wrong conclusions about intervention effects. Several quality components for the assessment of bias control have been suggested, but although they seem intrinsically valid, empirical evidence is needed to evaluate their effects on the extent and direction of bias. This narrative review summarizes the findings of methodological studies on the influence of bias in clinical trials. A number of methodological studies suggest that lack of adequate randomization in published trial reports may be associated with more positive estimates of intervention effects. The influence of double-blinding and follow-up is less clear. Several studies have found a significant association between funding sources and pro-industry conclusions. However, the methodological studies also show that bias is difficult to detect and appraise. The extent of bias in individual trials is unpredictable. A-priori exclusion of trials with certain characteristics is not recommended. Appraising bias control in individual trials is necessary to avoid making incorrect conclusions about intervention effects.

Recombinant human anti–transforming growth factor β1 antibody therapy in systemic sclerosis: A multicenter, randomized, placebo-controlled phase I/II trial of CAT-192

OBJECTIVE

To evaluate CAT-192, a recombinant human antibody that neutralizes transforming growth factor beta1 (TGFbeta1), in the treatment of early-stage diffuse cutaneous systemic sclerosis (dcSSc).

METHODS

Patients with SSc duration of <18 months were randomly assigned to the placebo group or to 1 of 3 CAT-192 treatment groups: 10 mg/kg, 5 mg/kg, 0.5 mg/kg. Infusions were given on day 0 and weeks 6, 12, and 18. The primary objective of this study was to evaluate the safety, tolerability, and pharmacokinetics of CAT-192. Secondary outcomes included the modified Rodnan skin thickness score (MRSS), the Scleroderma Health Assessment Questionnaire, assessment of organ-based disease, serum levels of soluble interleukin-2 receptor, collagen propeptides (N propeptide of type I [PINP] and type III collagen), and tissue levels of messenger RNA for procollagens I and III and for TGFbeta1 and TGFbeta2.

RESULTS

Forty-five patients were enrolled. There was significant morbidity and mortality, including 1 death in the group receiving 0.5 mg/kg of CAT-192 and 3 deaths in the group receiving 5 mg/kg of CAT-192. There were more adverse events and more serious adverse events in patients receiving CAT-192 than in those receiving placebo, although these events were not more frequent in the high-dose treatment group. The MRSS improved in all groups during the study, but there was no evidence of a treatment effect for CAT-192. Improvement in the MRSS correlated with the disease duration (r = -0.54, P = 0.0008). Changes in the PINP level from baseline correlated with changes in the MRSS (r = 0.37, P = 0.027).

CONCLUSION

We report the first evaluation of a systemically administered and repeatedly dosed anti-TGFbeta1 drug. In this pilot study, CAT-192, in doses up to 10 mg/kg, showed no evidence of efficacy. The utility of clinical and biochemical outcome measures and the feasibility of multicenter trials of early dcSSc were confirmed.

Mortality in systemic sclerosis: an international meta-analysis of individual patient data

PURPOSE

Studies on mortality associated with systemic sclerosis have been limited by small sample sizes. We aimed to obtain large-scale evidence on survival outcomes and predictors for this disease.

METHODS

We performed a meta-analysis of individual patient data from cohorts recruited from seven medical centers in the United States, Europe, and Japan, using standardized definitions for disease subtype and organ system involvement. The primary outcome was all-cause mortality. Standardized mortality ratios and predictors of mortality were estimated. The main analysis was based only on patients enrolled at each center within 6 months of diagnosis (incident cases).

RESULTS

Among 1645 incident cases, 578 deaths occurred over 11,521 person-years of follow-up. Standardized mortality ratios varied by cohort (1.5 to 7.2). In multivariate analyses that adjusted for age and sex, renal (hazard ratio [HR] = 1.9; 95% confidence interval [CI]: 1.4 to 2.5), cardiac (HR = 2.8; 95% CI: 2.1 to 3.8), and pulmonary (HR = 1.6; 95% CI: 1.3 to 2.2) involvement, and anti-topoisomerase I antibodies (HR = 1.3; 95% CI: 1.0 to 1.6), increased mortality risk. Renal, cardiac, and pulmonary involvement tended to occur together (P <0.001). For patients without adverse predictors for 3 years after enrollment, the subsequent risk of death was not significantly different from that for the general population in three cohorts, but was significantly increased in three cohorts that comprised mostly referred patients. Analyses that included all cases in each center (n = 3311; total follow-up: 19,990 person-years) yielded largely similar results.

CONCLUSION

Systemic sclerosis confers a high mortality risk, but there is considerable heterogeneity across settings. Internal organ involvement and anti-topoisomerase I antibodies are important determinants of mortality.

Randomized Trials of Neurosurgical Interventions: A Systematic Appraisal

OBJECTIVE

To systematically appraise the study design and quality of reporting of randomized controlled trials (RCTs) on neurosurgical procedures and to identify potential defects and biases.

METHODS

Randomized controlled trials with at least five patients comparing any neurosurgical procedure against another procedure, nonsurgical treatment, or no treatment were retrieved from MEDLINE, EMBASE, and the Cochrane Library. We analyzed study design, quality of reporting, and trial results.

RESULTS

The median sample size in the 108 eligible reports was 68 patients. Ninety-nine trials (91.7%) reported inclusion and exclusion criteria, 55 (50.9%) mentioned the randomization mode, and 87 (80.6%) adequately described withdrawals, but only 31 (28.7%) described allocation concealment, only 23 (21.3%) gave power calculations, and only 20 (18.5%) were adequately powered. Significant efficacy or trend for efficacy was claimed in 46 reports (42.6%), and no difference between the compared procedures was found in 60 trials (55.6%). Trials with a larger sample size were more likely to report withdrawals (P = 0.02) and power calculations (P = 0.006). Only 14 trials (13.6%) were double-blind, and this was less frequent in longer trials (P = 0.02). Among quality criteria, only the reporting of randomization mode improved significantly over time (P = 0.015).

CONCLUSION

Several aspects of the design and reporting of randomized controlled trials on neurosurgical procedures can be improved. Larger, adequately powered, and accurately reported trials are needed.