Methodological reporting quality of randomized controlled trials in three spine journals from 2010 to 2012

Systematic review of sample size calculations and reporting in randomized controlled trials in ophthalmology over a 20-year period

PURPOSE

Randomized Controlled Trials (RCTs) are considered the gold standard for the practice of evidence-based medicine. The purpose of this study is to systematically assess the reporting of sample size calculations in ophthalmology RCTs in 5 leading journals over a 20-year period. Reviewing sample size calculations in ophthalmology RCTs will shed light on the methodological quality of RCTs and, by extension, on the validity of published results.

METHODS

The MEDLINE database was searched to identify full reports of RCTs in the journals Ophthalmology, JAMA Ophthalmology, American Journal of Ophthalmology, Investigative Ophthalmology and Visual Science, and British Journal of Ophthalmology between January and December of the years 2000, 2010 and 2020. Screening identified 559 articles out of which 289 met the inclusion criteria for this systematic review. Data regarding sample size calculation reporting and trial characteristics was extracted for each trial by independent investigators.

RESULTS

In 2020, 77.9% of the RCTs reported sample size calculations as compared with 37% in 2000 (p < 0.001) and 60.7% in 2010 (p = 0.012). Studies reporting all necessary parameters for sample size recalculation increased significantly from 17.2% in 2000 to 39.3% in 2010 and 43.0% in 2020 (p < 0.001). Reporting of funding was greater in 2020 (98.8%) compared with 2010 (89.3%) and 2000 (53.1%). Registration in a clinical trials database occurred more frequently in 2020 (94.2%) compared to 2000 (1.2%; p < 0.001) and 2010 (68%; p < 0.001). In 2020, 38.4% of studies reported different sample sizes in the online registry from the published article. Overall, the most studied area in 2000 was glaucoma (29.6% of RCTs), whereas in 2010 and 2020, it was retina (40.2 and 37.2% of the RCTs, respectively). The number of patients enrolled in a study and the number of eyes studied was significantly greater in 2020 compared to 2000 and 2010 (p < 0.001).

CONCLUSION

Sample size calculation reporting in ophthalmology RCTs has improved significantly between the years 2000 and 2020 and is comparable to other fields in medicine. However, reporting of certain parameters remains inconsistent with current publication guidelines.

Network meta-analysis: What is its purpose in Orthopaedic literature?

Systematic reviews, of level-I primary literature, are the gold standard for the formation of Clinical Practice Guidelines in Orthopaedic Surgery. When systematic reviews have multiple groups of data, meta-analyses can be conducted to analyse the direct comparison of the data points (pairwise meta-analysis). Over recent years, statisticians have created a new statistical model called network meta-analyses that can be applied to systematic reviews. network meta-analyses allow for comparison of different treatment outcomes that may or may not have been directly assessed through level-I primary studies. network meta-analyses are appearing more and more in Orthopaedic Surgery literature; therefore, in this article, we discuss what a Network Meta-analysis is and its application in Orthopaedics.

Intervention Reporting of Published Trials Is Insufficient in Orthopaedic Surgery Journals: Application of the Template for Intervention Description and Replication Checklist

PURPOSE

Outcomes of randomized controlled trials (RCTs) have been shown to influence clinical decision making. Thus, the quality and reliability of these outcomes are essential for both patients and medical care providers. To date, no study has assessed the quality of intervention reporting of RCTs in orthopaedics. The aim of this study was to evaluate the quality of intervention reporting of published RCTs in the field of orthopaedics using the Template for Intervention Description and Replication (TIDieR) checklist.

METHODS

In this cross-sectional analysis, we applied the TIDieR checklist to assess the quality of intervention reporting in orthopaedic RCTs. Additionally, we evaluated the TIDieR checklist's influence on intervention reporting by comparing overall adherence to checklist items in trials published before the TIDieR checklist's release versus trials published after its release. Finally, we assessed whether certain factors were associated with the quality of intervention reporting.

RESULTS

From a random sample of 300 publications in orthopaedic journals, 175 parallel-arm and cluster RCTs were identified. The overall rate of adherence to TIDieR items was 58.4%. Only 31.4% of orthopaedic RCTs adhered to at least 6 of the 12 TIDieR checklist items, whereas 0% adhered to all 12 items. We found no significant improvement in the quality of intervention reporting in studies published after the TIDieR checklist's release compared with studies published before its release (P = .97). Additionally, preregistered trials were associated with more complete intervention reporting.

CONCLUSIONS

Our results suggest suboptimal reporting of orthopaedic RCT interventions. In addition, the TIDieR checklist's intended effect-to better the quality of RCT intervention reporting-appears to have fallen short of its goal.

CLINICAL RELEVANCE

Because outcomes of RCTs are used to guide clinical decision making, it is essential that orthopaedic surgeons and clinical practice guideline panels are equipped with high-quality published research. Increasing the accuracy of intervention reporting may lead to more accurate clinical application. Thus, adoption of more stringent reporting of trial interventions by researchers, authors, and journal editors may improve the quality of orthopaedic research, as well as improve patient outcomes.

Levels of evidence backing the AAOS clinical practice guidelines

Purpose:

The American Academy of Orthopaedic Surgeons produces clinical practice guidelines for the treatment of orthopedic injuries. We examined the strength of the evidence underlying these recommendations in order to answer the following questions: (1) Have AAOS work groups improved guideline creation practices to locate evidence to generate strong recommendations? (2) Is there variability in the available evidence based on anatomic site or stage of care? (3) Has the level of evidence supporting improved over time?

Methods:

Twenty-two current guidelines of the Academy were examined which yielded 408 individual recommendations. These recommendations were assigned one of five strength of evidence ratings (strong, moderate, limited, inconclusive, consensus) by the guideline panel, based on the availability and quality of the supporting evidence. From these guidelines, we extracted all of the recommendations and their corresponding evidence ratings. We then classified the recommendations by stage of care, year, and anatomical site.

Results:

The distribution of the levels of evidence was as follows: 77 (18.9%) were based on consensus; 53 (13.0%) were inconclusive; 93 (22.8%) were based on limited evidence; 112 (27.5%) were based on moderate evidence; and 73 were based on (17.9%) strong evidence. Strong strength of evidence was found in 45.2% of the recommendations for preventive/screening/diagnostic care, 41.1% of nonsurgical treatment, 45.1% of surgical treatment, 51.1% of rehabilitation/postoperative treatment, and 45.5% of the recommendations that had mixed stages of care. Inconclusive strength of evidence was found to be prevalent from 2009–2013, but was eliminated starting in 2014.

Conclusions:

Only 73 (17.9%) recommendations generated by the Academy in its 22 clinical practice guidelines are based on a “strong” strength of evidence. More robust research is needed in orthopedics to bolster confidence in the recommendations in future guideline updates.

Trends and predictors of biomedical research quality, 1990-2015: a meta-research study

OBJECTIVE

To measure the frequency of adequate methods, inadequate methods and poor reporting in published randomised controlled trials (RCTs) and test potential factors associated with adequacy of methods and reporting.

DESIGN

Retrospective analysis of RCTs included in Cochrane reviews. Time series describes the proportion of RCTs using adequate methods, inadequate methods and poor reporting. A multinomial logit model tests potential factors associated with methods and reporting, including funding source, first author affiliation, clinical trial registration status, study novelty, team characteristics, technology and geography.

DATA

Risk of bias assessments for random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data and selective reporting, for each RCT, were mapped to bibliometric and funding data.

OUTCOMES

Risk of bias on six methodological dimensions and RCT-level overall assessment of adequate methods, inadequate methods or poor reporting.

RESULTS

This study analysed 20 571 RCTs. 5.7% of RCTs used adequate methods (N=1173). 59.3% used inadequate methods (N=12 190) and 35.0% were poorly reported (N=7208). The proportion of poorly reported RCTs decreased from 42.5% in 1990 to 30.2% in 2015. The proportion of RCTs using adequate methods increased from 2.6% in 1990 to 10.3% in 2015. The proportion of RCTs using inadequate methods increased from 54.9% in 1990 to 59.5% in 2015. Industry funding, top pharmaceutical company affiliation, trial registration, larger authorship teams, international teams and drug trials were associated with a greater likelihood of using adequate methods. National Institutes of Health funding and university prestige were not.

CONCLUSION

Even though reporting has improved since 1990, the proportion of RCTs using inadequate methods is high (59.3%) and increasing, potentially slowing progress and contributing to the reproducibility crisis. Stronger incentives for the use of adequate methods are needed.

Trustworthiness of randomized trials in endocrinology-A systematic survey

Background

Trustworthy (i.e. low risk of bias) randomized clinical trials (RCTs) play an important role in evidence-based decision making. We aimed to systematically assess the risk of bias of trials published in high-impact endocrinology journals.

Methods

We searched the MEDLINE/PubMed database between 2014 and 2016 for phase 2–4 RCTs evaluating endocrine-related therapies. Reviewers working independently and in duplicate used the Cochrane Risk of Bias Tool (CCRBT) to determine the extent to which the methods reported protected the results of each RCT from bias.

Results

We assessed 292 eligible RCTs, of which 40% (116) were judged to be at low risk, 43% (126) at moderate, and 17% (50) at high risk of bias. Blinding of outcome assessment was the least common domain reported 43% (125), while selective reporting of outcomes was the most common 97% (282). In multivariable analysis, RCTs with a parallel design (OR 2.4; 95% CI; 1.2–4.6) and funded by for-profit sources (OR 2.2; 95% CI; 1.3–3.6) were more likely to be at low risk of bias.

Conclusions

Trustworthy evidence should ultimately shape care to improve the likelihood of desirable patient outcomes. Six out-of 10 RCTs published in top endocrine journals are at moderate/high-risk of bias. Improving this should be a priority in endocrine research.

An Evaluation of Reporting Guidelines and Clinical Trial Registry Requirements Among Orthopaedic Surgery Journals

BACKGROUND

The responsibility for ensuring that studies are adequately reported is primarily that of those conducting the study; however, journal policies may influence how thoroughly authors choose to report their research. The use of reporting guidelines and prospective trial registration are promising avenues for ensuring that published studies adhere to the highest methodological standards. The purpose of this study is to evaluate orthopaedic surgery journal policies regarding reporting guidelines and trial registration, and to evaluate the effects that these policies have on adherence to reporting.

METHODS

We conducted a cross-sectional survey of journal policies and "Instructions for Authors" to determine the journals' policies and guidance regarding use of reporting guidelines and study registration. We also examined whether trials published in journals referencing CONSORT (Consolidated Standards of Reporting Trials) had higher rates of compliance with publishing a CONSORT flow diagram and whether journals with trial registration policies were more likely to contain registered trials than journals without these requirements.

RESULTS

Of the 21 orthopaedic surgery journals, 6 (29%) did not mention a single guideline, and clinical trial registration was required by 11 (52%) orthopaedic surgery journals and recommended by 2 (10%). Of the 21 general medical journals, 3 (14%) did not mention a single guideline, and trial registration was required by 13 (62%) general medical journals and recommended by 5 (24%) others. Furthermore, journals that referenced CONSORT were more likely to publish trials with a CONSORT flow diagram. Journals with trial registration policies were more likely to publish registered trials.

CONCLUSIONS

Reporting guidelines and trial registration are suboptimally required or recommended by orthopaedic surgery journals. These 2 mechanisms may improve methodology and quality, and should be considered for adoption by journal editors in orthopaedic surgery.

CLINICAL RELEVANCE

Because orthopaedic surgeons rely on high-quality research to direct patient care, measures must be taken to ensure that published research is of the highest quality. The use of reporting guidelines and prospective clinical trial registration may improve the quality of orthopaedic research, thereby improving patient care.

Assessing the quality of reports about randomized controlled trials of scalp acupuncture combined with another treatment for stroke

BACKGROUND

This study was designed to assess the quality of reporting on randomized controlled trials (RCTs) of scalp acupuncture for the treatment of stroke.

METHODS

The following 8 databases were systematically investigated from their inception to December 2015: PubMed, Embase, Cochrane Library, China National Knowledge Infrastructure, National Institute of Informatics Scholarly and Academic Information Navigator, National Digital Science Library, Korean Traditional Knowledge Portal, and Korean Studies Information Service System. RCTs utilizing scalp acupuncture as an intervention for stroke were selected, and the quality of reports was assessed based on the Consolidated Standards of Reporting Trials 2010 statement (CONSORT) and Standards for Reporting Interventions in Controlled Trials of Acupuncture 2010 (STRICTA). For each study, the overall quality score (OQS) of 13 CONSORT items, a combined key methodological index score (MIS) of 5 CONSORT items, and the OQS of 17 STRICTA items were measured.

RESULTS

The original reports of 63 RCTs were ultimately obtained, and the median CONSORT OQS was 7 (minimum 2, maximum 11). Particularly, the items 'trial design', 'sample size', 'ancillary analyses', and 'harms' had a positive rate of less than 10%. The median MIS was 1 (minimum 0, maximum 5), with 'allocation concealment and implementation' and 'intent-to-treat analysis (ITT) analysis' having a positive rate of less than 10%. The median STRICTA OQS was 11 (minimum 6, maximum 14), and only the items 'sample size' and 'intent-to-treat analysis' were reported, with a positive rate of less than 10%. The mean CONSORT OQS increased by approximately 0.81 for each 5-year period in which manuscripts were published (95% confidence interval: 0.43 to 1.19; p < 0.001). No variable was significantly associated with MIS in the ordinal regression model.

CONCLUSION

The quality of reports on RCTs investigating scalp acupuncture treatment for stroke was moderate to low. Furthermore, reporting of some items was either insufficient or inadequate in the majority of studies. In order to improve and standardize the quality of RCTs investigating scalp acupuncture for stroke, CONSORT and STRICTA guidelines should be utilized more frequently.

Assessing the quality of reports about randomized controlled trials of scalp acupuncture treatment for vascular dementia

BACKGROUND

This study aimed to evaluate the quality of reports about randomized controlled trials (RCTs) of scalp acupuncture (SA) for the treatment of vascular dementia (VD).

METHOD

A systematic search of reports published through to December 2015 was performed in eight databases. The quality of RCTs that used SA as an intervention for VD was evaluated based on the 2010 Consolidated Standards for Reporting of Trials (CONSORT) and 2010 Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) guidelines. Thirteen items from the CONSORT guideline were scored to give an overall quality score (OQS, range 0-13), and a combined key methodological index score (MIS) (range 0-5) of five key methodological items was measured. The OQS of 17 items from the STRICTA guideline (range 0-17) was also measured.

RESULTS

In total, 26 reports were evaluated. The median OQS based on the CONSORT guideline was 8 (minimum 5, maximum 11), and "trial design," "sample size," "ancillary analyses," and "harms" had a positive rate of less than 10%. The median MIS was 2 (minimum 0, maximum 5), with "allocation concealment and implementation," "blinding," and "intent-to-treat analysis" having a positive rate of less than 15%. The median OQS based on the STRICTA guideline was 12 (minimum 8, maximum 14), with "extent to which treatment was varied (1c)," "number of needle insertions per subject per session (2a)," and "setting and context of treatment (4b)" having a positive rate of less than 10%.

CONCLUSIONS

The overall quality of reports on RCTs of SA treatment for VD was moderate to low. The quality of methodological items was markedly lower than that of other items. The CONSORT and STRICTA guidelines should be used more frequently to standardize the quality of RCT reports of SA treatment for VD.

Methodological Reporting Quality of Randomized Controlled Trials in 3 Leading Diabetes Journals From 2011 to 2013 Following CONSORT Statement: A System Review

To appraise the current reporting methodological quality of randomized clinical trials (RCTs) in 3 leading diabetes journals.We systematically searched the literature for RCTs in Diabetes Care, Diabetes and Diabetologia from 2011 to 2013.Characteristics were extracted based on Consolidated Standards of Reporting Trials (CONSORT) statement. Generation of allocation, concealment of allocation, intention-to-treat (ITT) analysis and handling of dropouts were defined as primary outcome and "low risk of bias." Sample size calculation, type of intervention, country, number of patients, funding source were also revealed and descriptively reported. Trials were compared among journals, study years, and other characters.A total of 305 RCTs were enrolled in this study. One hundred eight (35.4%) trials reported adequate generation of allocation, 87 (28.5%) trials reported adequate concealment of allocation, 53 (23.8%) trials used ITT analysis, and 130 (58.3%) trials were adequate in handling of dropouts. Only 15 (4.9%) were "low risk of bias" trials. Studies at a large scale (n > 100) or from European presented with more "low risk of bias" trials than those at a small scale (n ≤ 100) or from other regions. No improvements were found in these 3 years.This study shows that methodological reporting quality of RCTs in the major diabetes journals remains suboptimal. It can be further improved to meet and keep up with the standards of the CONSORT statement.

Comparison of methodological quality of positive versus negative comparative studies published in Indian medical journals: a systematic review

OBJECTIVES

Published negative studies should have the same rigour of methodological quality as studies with positive findings. However, the methodological quality of negative versus positive studies is not known. The objective was to assess the reported methodological quality of positive versus negative studies published in Indian medical journals.

DESIGN

A systematic review (SR) was performed of all comparative studies published in Indian medical journals with a clinical science focus and impact factor >1 between 2011 and 2013. The methodological quality of randomised controlled trials (RCTs) was assessed using the Cochrane risk of bias tool, and the Newcastle-Ottawa scale for observational studies. The results were considered positive if the primary outcome was statistically significant and negative otherwise. When the primary outcome was not specified, we used data on the first outcome reported in the history followed by the results section. Differences in various methodological quality domains between positive versus negative studies were assessed by Fisher's exact test.

RESULTS

Seven journals with 259 comparative studies were included in this SR. 24% (63/259) were RCTs, 24% (63/259) cohort studies, and 49% (128/259) case-control studies. 53% (137/259) of studies explicitly reported the primary outcome. Five studies did not report sufficient data to enable us to determine if results were positive or negative. Statistical significance was determined by p value in 78.3% (199/254), CI in 2.8% (7/254), both p value and CI in 11.8% (30/254), and only descriptive in 6.3% (16/254) of studies. The overall methodological quality was poor and no statistically significant differences between reporting of methodological quality were detected between studies with positive versus negative findings.

CONCLUSIONS

There was no difference in the reported methodological quality of positive versus negative studies. However, the uneven reporting of positive versus negative studies (72% vs 28%) indicates a publication bias in Indian medical journals with an impact factor of >1.

Quality of reporting on randomized controlled trials on recurrent spontaneous abortion in China

Background

Despite increasing numbers of RCTs done in China, detailed information on the quality of Chinese RCTs is still missing. The aim of this study was to assess the reporting quality of RSA RCTs and to identify significant predictors of reporting quality.

Methods

A literature review was conducted with the aim of identifying published RCTs on RSA conducted in China. In order to rate the report quality, we scored 1 for the item of CONSORT 2010 if it was reported and 0 if it was not stated or unclear. An overall quality score (OQS) with a range of 0–15 and a key methodological index score (MIS) with a range of 0–3 were calculated for each trial.

Results

A total of 98 relevant RCTs were included in the final analysis. The median OQS was 7, with a minimum of 1 and maximum of 12. The general level of OQS was not high, especially among ‘sample size,’ ‘baseline data,’ ‘outcomes and estimation,’ and ‘ancillary analyses,’ all of which had a positive rate of less than 10%. The median MIS was 1 with a minimum of 0 and maximum of 1. ‘Allocation concealment,’ ‘blinding,’ and ‘intention-to-treat analysis’ were mentioned in 1 (1%), 1 (1%) and 69 (70%) of the studies, respectively. In univariate analysis, funding was the only factor associated with an increased OQS. Specifically, the mean OQS increased by approximately 1.52 for manuscripts supported by funding (95% CI: 0.12 – 2.92; p = 0.03). With regard to the MIS, no association was found for any variable.

Conclusion

RCTs of RSA conducted in China need improvement in order to meet the level of “reporting quality” required by the CONSORT statement.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-015-0665-6) contains supplementary material, which is available to authorized users.