Improvement in the Quality of Randomized Controlled Trials Among General Anesthesiology Journals 2000 to 2006: A 6-Year Follow-Up

Blinding, sham, and treatment effects in randomized controlled trials for back pain in 2000-2019: A review and meta-analytic approach

BACKGROUND

Blinding aims to minimize biases from what participants and investigators know or believe. Randomized controlled trials, despite being the gold standard to evaluate treatment effect, do not generally assess the success of blinding. We investigated the extent of blinding in back pain trials and the associations between participant guesses and treatment effects.

METHODS

We did a review with PubMed/OvidMedline, 2000-2019. Eligibility criteria were back pain trials with data available on treatment effect and participants' guess of treatment. For blinding, blinding index was used as chance-corrected measure of excessive correct guess (0 for random guess). For treatment effects, within- or between-arm effect sizes were used. Analyses of investigators' guess/blinding or by treatment modality were performed exploratorily.

RESULTS

Forty trials (3899 participants) were included. Active and sham treatment groups had mean blinding index of 0.26 (95% confidence interval: 0.12, 0.41) and 0.01 (-0.11, 0.14), respectively, meaning 26% of participants in active treatment believed they received active treatment, whereas only 1% in sham believed they received sham treatment, beyond chance, that is, random guess. A greater belief of receiving active treatment was associated with a larger within-arm effect size in both arms, and ideal blinding (namely, "random guess," and "wishful thinking" that signifies both groups believing they received active treatment) showed smaller effect sizes, with correlation of effect size and summary blinding indexes of 0.35 (p = 0.028) for between-arm comparison. We observed uniformly large sham treatment effects for all modalities, and larger correlation for investigator's (un)blinding, 0.53 (p = 0.046).

CONCLUSION

Participants in active treatments in back pain trials guessed treatment identity more correctly, while those in sham treatments tended to display successful blinding. Excessive correct guesses (that could reflect weaker blinding and/or noticeable effects) by participants and investigators demonstrated larger effect sizes. Blinding and sham treatment effects on back pain need due consideration in individual trials and meta-analyses.

Methodologies for systematic reviews with meta-analysis of randomised clinical trials in pain, anaesthesia, and perioperative medicine

Systematic reviews and meta-analyses (SRMAs) are increasing in popularity, but should they be used to inform clinical decision-making in anaesthesia? We present evidence that the certainty of evidence from SRMAs in anaesthesia (and in general) may be unacceptably low because of risks of bias exaggerating treatment effects, unexplained heterogeneity reducing certainty in estimates, random errors, and widespread prevalence of publication bias. We also present the latest methodological advances to help improve the certainty of evidence from SRMAs. The target audience includes both review authors and practising clinicians to help with SRMA appraisal. Issues discussed include minimising risks of bias from included trials, trial sequential analysis to reduce random error, updated methods for presenting effect estimates, and novel publication bias tests for commonly used outcome measures. These methods can help to reduce spurious conclusions on clinical significance, explain statistical heterogeneity, and reduce false positives when evaluating small-study effects. By reducing concerns in these domains of Grading of Recommendations, Assessment, Development and Evaluation, it should help improve the certainty of evidence from SRMAs used for decision-making in anaesthesia, pain, and perioperative medicine.

Detecting the extent of control over selection bias relating to oral health and otorhinolaryngology: cross-sectional study

BACKGROUND

The authors of randomized controlled trials will usually claim that they have met the randomization process criterion. However, sequence generation schemes differ and some schemes that are claimed to be randomized are not genuinely randomized. Even less well understood, and often more difficult to ascertain, is whether the allocation was really concealed.

OBJECTIVE

To detect the extent of control over selection bias, in a comparison between two Cochrane groups: oral health and otorhinolaryngology; and to describe the methods used to control for this bias.

DESIGN AND SETTING

Cross-sectional study conducted in a public university in São Paulo, Brazil.

METHODS

The risk of selection bias in 1,714 records indexed in Medline database up to 2018 was assessed, independent of language and access. Two dimensions implicated in the allocation were considered: generation of the allocation sequence; and allocation concealment.

RESULTS

We included 420 randomized controlled trials and all of them were evaluated to detect selection bias. In the sample studied, only 28 properly controlled the selection bias. Lack of control over selection bias was present in 80% of the studies evaluated in both groups.

CONCLUSION

The two groups were similar regarding control over selection bias. They are also similar to the methods used. The dimension of allocation concealment appears to be a limiting factor with regard to production of randomized controlled trials with low risk of selection bias. The quality of reporting in studies on oral health and otorhinolaryngology is suboptimal and needs to be improved, in line with other fields of healthcare.

Impact of the Consolidated Standards of Reporting Trials (CONSORT) checklist on reporting of randomized clinical trials in traditional Chinese medicine

OBJECTIVE

To examine the reporting characteristics and key methodological factors of randomized controlled trials (RCTs) in the field of traditional Chinese medicine and assess whether use of the Consolidated Standards of Reporting Trials (CONSORT) statement is associated with improvement in the quality of reports of RCTs.

METHOD

RCTs published in two major traditional Chinese medicine journals were retrieved and were systematically reviewed. Of those journals, one adopted the CONSORT statement (Journal of Chinese Integrative Medicine (JCIM)) and one did not (Chinese Journal of Integrative Medicine (CJIM)). Items within the CONSORT 2001 and the CONSORT 2010 statements were taken to develop a 63-item coding manual. The Jadad scale was also used to assess methodological quality of RCTs. Two observers assessed the reporting of reports and extracted data independently. P < 0.05 was considered statistically significant. SPSS was used for all analyses.

RESULTS

A total of 76 RCTs were included for the analysis (38 from JCIM and 38 from CJIM). Significant improvements both in the overall quality of reporting (CONSORT score) and reporting of methodological items (Jadad scale) over time were observed in adopter journal (JCIM) (P = 0.001). Overall CONSORT score of studies published during 2010 to 2011 (post-CONSORT) increased 15.30 averagely with 95%CI 8.34 to 22.26 (P = 0.001). Of those, items addressed on reporting of methodological quality were different significantly too. JCIM had more 1.30 items addressed than CJIM in average (95% CI 0.52 to 2.08, P = 0.003). In the multivariate analysis regression, the page length and CONSORT-promoting status have statistically significant associated with the (P = 0.001 and P = 0.040, respectively).

CONCLUSION

Although a large room needs to improve the reporting of randomized clinical trials in traditional Chinese medicine, the impact for improvement of reporting of TCM RCTs has been proven in some extent.

Assessing blinding in trials of psychiatric disorders: a meta-analysis based on blinding index

The assessment of blinding in RCTs is rarely performed. Currently most studies that do report data on evaluation of blinding merely report percentages of correct guessing, not taking into account correct guessing by chance. Blinding assessment using the blinding index (BI) has never been performed in a systematic review on studies of major psychiatric disorders. This study is a systematic review of psychiatric randomized control trials using the BI as a chance-corrected measurement of blinding, a tool to analyze and understand the patterns of blinding across studies of major psychiatric disorders with available data. Of 2467 psychiatric RCTs from 2000 to 2010, 66 reported on blinding and 40 studies were found to have enough information on evaluation of blinding to be analyzed using the BI. The experimental treatment groups had an average BI value of 0.14 and the control groups had an average BI value of 0.00. The most common BI scenario was random-random, indicating ideal blinding. A positive correlation between effect size and more correct guesses was also found. Overall, based on BI values and the most common blinding scenario, the published articles on major psychiatric disorders from 2000 to 2010, which reported on blinding assessment for patients, were effectively blinded.

A look at the potential association between PICOT framing of a research question and the quality of reporting of analgesia RCTs

BACKGROUND

Methodologists have proposed the formation of a good research question to initiate the process of developing a research protocol that will guide the design, conduct and analysis of randomized controlled trials (RCTs), and help improve the quality of reporting such studies. Five constituents of a good research question based on the PICOT framing include: Population, Intervention, Comparator, Outcome, and Time-frame of outcome assessment. The aim of this study was to analyze if the presence a structured research question, in PICOT format, in RCTs used within a 2010 meta-analysis investigating the effectiveness of femoral nerve blocks after total knee arthroplasty, is independently associated with improved quality of reporting.

METHODS

Twenty-three RCT reports were assessed for the quality of reporting and then examined for the presence of the five constituents of a structured research question based on PICOT framing. We created a PICOT score (predictor variable), with a possible score between 0 and 5; one point for every constituent that was included. Our outcome variable was a 14 point overall reporting quality score (OQRS) and a 3 point key methodological items score (KMIS) based on the proper reporting of allocation concealment, blinding and numbers analysed using the intention-to-treat principle. Both scores, OQRS and KMIS, are based on the Consolidated Standards for Reporting Trials (CONSORT) statement. A multivariable regression analysis was conducted to determine if PICOT score was independently associated with OQRS and KMIS.

RESULTS

A completely structured PICOT score question was found in 2 of the 23 RCTs evaluated. Although not statistically significant, higher PICOT was associated with higher OQRS [IRR: 1.267; 95% confidence interval (CI): 0.984, 1.630; p = 0.066] but not KMIS (1.061 (0.515, 2.188); 0.872). These results are comparable to those from a similar study in terms of the direction and range of IRRs estimates. The results need to be interpreted cautiously due to the small sample size.

CONCLUSIONS

This study showed that PICOT framing of a research question in anesthesia-related RCTs is not often followed. Even though a statistically significant association with higher OQRS was not found, PICOT framing of a research question is still an important attribute within all RCTs.

Trends in worldwide volume and methodological quality of surgical randomized controlled trials

OBJECTIVE

To assess worldwide trends in volume and methodological quality of published surgical randomized controlled trials (RCTs) over the past decade.

BACKGROUND

Randomized controlled trials are essential for clinical decision making. It has repeatedly been suggested that surgical RCTs are scarce and of mediocre quality.

METHODS

We systematically searched PubMed for surgical RCTs published in 1999 and 2009. Characteristics and risks of bias were extracted. Trials where compared between study years and geographical regions. Primary outcome was "low risk of bias," defined by all of the following: adequate allocation generation and concealment, intention-to-treat analysis, and adequate dropout handling.

RESULTS

The volume of published surgical RCTs increased by 50%, from 300 in 1999 to 450 in 2009. Volume increased in Europe (27% increase), Asia/Oceania (160% increase), and Africa/South America (416% increase) but decreased in North America (23% decrease), although the United States remained the country with the highest number of published RCTs. In 2009, methodological quality of surgical trials improved in terms of sample size calculation, adequate generation of randomization sequence, concealment of randomization sequence, and use of intention-to-treat analysis as compared with 1999 (P < 0.001 for all). The proportion of low risk of bias trials increased from 6% to 14% (prevalence ratio 2.59; 95% confidence interval 1.55-4.32). In 2009, the highest proportion of low risk of bias trials was from Europe (23%), whereas the lowest was from Asia/Oceania (5%).

CONCLUSIONS

Volume and quality of surgical RCTs improved although striking differences exist between continents and countries. Structured education in trial methodology, enforced adherence to existing guidelines, and improved research infrastructure may guide further improvements.

The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement

BACKGROUND

Randomized controlled trials (RCTs) are routinely used in systematic reviews and meta-analyses that help inform healthcare and policy decision making. The proper reporting of RCTs is important because it acts as a proxy for health care providers and researchers to appraise the quality of the methodology, conduct and analysis of an RCT. The aims of this study are to analyse the overall quality of reporting in 23 RCTs that were used in a meta-analysis by assessing 3 key methodological items, and to determine factors associated with high quality of reporting. It is hypothesized that studies with larger sample sizes, that have funding reported, that are published in journals with a higher impact factor and that are in journals that have adopted or endorsed the CONSORT statement will be associated with better overall quality of reporting and reporting of key methodological items.

METHODS

We systematically reviewed RCTs used within an anesthesiology related post-operative pain management meta-analysis. We included all of the 23 RCTs used, all of which were parallel design that addressed the use of femoral nerve block in improving outcomes after total knee arthroplasty. Data abstraction was done independently by two reviewers. The two main outcomes were: 1) 15 point overall quality of reporting score (OQRS) based on the Consolidated Standards for Reporting Trials (CONSORT) and 2) 3 point key methodological item score (KMIS) based on allocation concealment, blinding and intention-to-treat analysis.

RESULTS

Twenty-three RCTs were included. The median OQRS was 9.0 (Interquartile Range = 3). A multivariable regression analysis did not show any significant association between OQRS or KMIS and our four predictor variables hypothesized to improve reporting. The direction and magnitude of our results when compared to similar studies suggest that the sample size and impact factor are associated with improved key methodological item reporting.

CONCLUSIONS

The quality of reporting of RCTs used within an anesthesia related meta-analysis is poor to moderate. The information gained from this study should be used by journals to register the urgency for RCTs to be clear and transparent in reporting to help make literature accessible and comparable.

Intranasal fentanyl in the treatment of acute pain--a systematic review

Due to its non-invasive mode of administration, intranasal (IN) application of drugs may be a valuable alternative to non-invasive pain management. With characteristics that appear to be ideal for IN application, IN fentanyl may have a place in the out-of-hospital treatment and the paediatric population. The objective of this systematic review was to evaluate the current evidence of IN fentanyl in the treatment of acute pain. Reports of randomized controlled trials (RCTs) of IN fentanyl in treatment of pain were systematically sought using the PubMed database, Embase, Google scholar, Cochrane database, and Cumulative Index to Nursing and Allied Health Literature. Reports were considered for inclusion if they were double-blinded randomized controlled trials (RCTs) of IN fentanyl in the treatment of acute pain. Thirty-two RCTs were identified, and 16 were included in the final analysis. No significant analgesic differences between IN fentanyl and intravenous (IV) fentanyl were demonstrated in treatment of acute and post-operative pain. Significant analgesic effect of IN fentanyl was demonstrated in the treatment of breakthrough pain in cancer patients. In the paediatric population, results demonstrated some analgesic effect of IN fentanyl following myringotomy, no analgesic effect following voiding cystourethrography, and finally, no significant analgesic difference after long bone fractures, in burns patients, and in post-operative pain relief when compared to IV morphine, oral morphine, or IV fentanyl, respectively. Significant analgesic effect of IN fentanyl was demonstrated in the treatment of breakthrough pain in cancer patients. However, the significant deficiencies in trials investigating acute and post-operative pain, and the paediatric population makes firm recommendations impossible.

Reporting of methods was better in the Clinical Trials Registry-India than in Indian journal publications

OBJECTIVE

We sought to evaluate if editorial policies and the reporting quality of randomized controlled trials (RCTs) had improved since our 2004-05 survey of 151 RCTs in 65 Indian journals, and to compare reporting quality of protocols in the Clinical Trials Registry-India (CTRI).

STUDY DESIGN AND SETTING

An observational study of endorsement of Consolidated Standards for the Reporting of Trials (CONSORT) and International Committee of Medical Journal Editors (ICMJE) requirements in the instructions to authors in Indian journals, and compliance with selected requirements in all RCTs published during 2007-08 vs. our previous survey and between all RCT protocols in the CTRI on August 31, 2010 and published RCTs from both surveys.

RESULTS

Journal policies endorsing the CONSORT statement (22/67, 33%) and ICMJE requirements (35/67, 52%) remained suboptimal, and only 4 of 13 CONSORT items were reported in more than 50% of the 145 RCTs assessed. Reporting of ethical issues had improved significantly, and that of methods addressing internal validity had not improved. Adequate methods were reported significantly more frequently in 768 protocols in the CTRI, than in the 296 published trials.

CONCLUSION

The CTRI template facilitates the reporting of valid methods in registered trial protocols. The suboptimal compliance with CONSORT and ICMJE requirements in RCTs published in Indian journals reduces credibility in the reliability of their results.

Has the quality of abstracts for randomised controlled trials improved since the release of Consolidated Standards of Reporting Trial guideline for abstract reporting? A survey of four high-profile anaesthesia journals

BACKGROUND AND OBJECTIVE

Randomised controlled trial (RCT) abstracts published in journal articles have traditionally been deficient of crucial information. To improve the quality of RCT abstracts, in January 2008, the Consolidated Standards of Reporting Trial (CONSORT) group published a checklist of essential information for inclusion. The current study assessed whether there has been an improvement in the quality of RCT abstracts published in main anaesthesia journals since this new guideline was introduced.

METHODS

Articles involving human RCTs published in four high-profile anaesthesia journals (Anaesthesia, Anesthesia & Analgesia, Anesthesiology and the European Journal of Anaesthesiology) were reviewed, comparing those published from October 2005 to September 2006 (pre-CONSORT abstracts) with those published from October 2008 to September 2009 (post-CONSORT abstracts). Trials involving healthy volunteers or cadavers, cost-effectiveness studies, meta-analyses and letters were excluded. Abstracts from remaining RCTs were randomly assigned to four reviewers in a blinded fashion and reviewed for content using the new CONSORT checklist.

RESULTS

In total, 527 RCT abstracts (pre-CONSORT RCTs, n = 275 and post-CONSORT RCTs, n = 252) were analysed. The majority of abstracts in both groups provided an appropriate description of study interventions (73.1 and 73.8%, pre-CONSORT abstracts versus post-CONSORT abstracts, respectively), objective (91.3 and 90.1%) and conclusions (72.4 and 66.3%). From pre-CONSORT to post-CONSORT guidelines for abstract reporting, there were significant improvements in correctly identifying blinding (18.2-29%) and harmful effects (31.6-42.1%). The improvement in reporting the nature of the trial in abstract titles (20.1-29%) and primary outcome measure in the methods section (22.9-30.6%) did not reach significance. There was no clear improvement in the already poor reporting of trial design, participants, randomisation, recruitment, outcomes, trial registration and funding sources.

CONCLUSIONS

Despite some promising improvements and inter-journal differences, the overall quality of RCT abstracts and adherence to the CONSORT checklist for abstracts remains poor.

Inconsistent survey reporting in anesthesia journals

BACKGROUND

As with other types of research, there are concerns about reporting of survey research in anesthesia journals. We hypothesized that use of survey reporting items would be inconsistent in survey research reported in anesthesia journals.

METHODS

After a literature review we constructed a 17-item reporting list for a limited systematic review of survey reporting in 6 anesthesia journals. We identified survey reports by MEDLINE (PubMed) search for January 2000 to April 2009.

RESULTS

The initial search identified 347 publications. Of these, we excluded 107 because they were not questionnaire surveys (often audits), were reviews, or were letters. We therefore identified 240 surveys published as full survey reports. From the 17-item reporting list, the median number of items recorded was 9 (interquartile range: 7 to 10; range 2 to 15). The number (and percentage) of surveys reporting specific items ranged widely for different items: from 9 surveys (4%; 95% confidence interval [CI]: 2% to 7%) for sample size to 240 surveys (100%; 95% CI: 98% to 100%) for response rate. In addition to sample size, the 5 least frequently reported items included the following: reporting confidence intervals, 21 surveys (9%; 95% CI: 6% to 13%); stating a hypothesis, 23 of 240 surveys (10%; 95% CI: 7% to 14%); accounting for nonresponders, 61 surveys (25%; 95% CI: 20% to 31%); and survey design, 67 surveys (28%; 95% CI: 33% to 34%).

CONCLUSIONS

Inconsistent reporting may compromise the transparency and reproducibility of survey reports.

The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed

OBJECTIVES

To examine the reporting characteristics and methodological details of randomised trials indexed in PubMed in 2000 and 2006 and assess whether the quality of reporting has improved after publication of the Consolidated Standards of Reporting Trials (CONSORT) Statement in 2001.

DESIGN

Comparison of two cross sectional investigations. Study sample All primary reports of randomised trials indexed in PubMed in December 2000 (n=519) and December 2006 (n=616), including parallel group, crossover, cluster, factorial, and split body study designs.

MAIN OUTCOME MEASURES

The proportion of general and methodological items reported, stratified by year and study design. Risk ratios with 95% confidence intervals were calculated to represent changes in reporting between 2000 and 2006.

RESULTS

The majority of trials were two arm (379/519 (73%) in 2000 v 468/616 (76%) in 2006) parallel group studies (383/519 (74%) v 477/616 (78%)) published in specialty journals (482/519 (93%) v 555/616 (90%)). In both 2000 and 2006, a median of 80 participants were recruited per trial for parallel group trials. The proportion of articles that reported drug trials decreased between 2000 and 2006 (from 393/519 (76%) to 356/616 (58%)), whereas the proportion of surgery trials increased (51/519 (10%) v 128/616 (21%)). There was an increase between 2000 and 2006 in the proportion of trial reports that included details of the primary outcome (risk ratio (RR) 1.18, 95% CI 1.04 to 1.33), sample size calculation (RR 1.66, 95% CI 1.40 to 1.95), and the methods of random sequence generation (RR 1.62, 95% CI 1.32 to 1.97) and allocation concealment (RR 1.40, 95% CI 1.11 to 1.76). There was no difference in the proportion of trials that provided specific details on who was blinded (RR 0.91, 95% CI 0.75 to 1.10).

CONCLUSIONS

Reporting of several important aspects of trial methods improved between 2000 and 2006; however, the quality of reporting remains well below an acceptable level. Without complete and transparent reporting of how a trial was designed and conducted, it is difficult for readers to assess its conduct and validity.