Clinical trials in general surgical journals: Are methods better reported?

An Assessment of Randomized Controlled Trial Quality in The Journal of Bone & Joint Surgery: Update from 2001 to 2013

BACKGROUND

The quality of reporting of randomized controlled trials (RCTs) published in The Journal of Bone & Joint Surgery (JBJS) from 1988 to 2000 was previously analyzed. The purpose of this current study was to analyze the quality of reporting of RCTs published in JBJS from 2001 to 2013 to identify trends over time and potential areas of improvement for future clinical trials.

METHODS

A manual search of the JBJS database identified RCTs published between January 2001 and December 2013. Quality assessments, using the Detsky quality-of-reporting index (Detsky score), a modified Cochrane risk-of-bias tool, and abstraction of relevant data identifying predictors of quality, were conducted.

RESULTS

A total of 5,780 publications were identified in JBJS from 2001 to 2013, with 285 RCTs (4.9%), representing an increase from the prior 13-year period. The overall mean transformed Detsky score (and standard error) increased significantly (p < 0.001) from 68.1% ± 1.67% to 76.24% ± 0.72%. The percentage of multicenter RCTs decreased from 67% to 31%. The percentage of positive trials also decreased from 80% to 50.5%, as did the mean sample size (212 to 166). Regression analysis indicated that trials with an epidemiologist as the first author and nonsurgical trials were significantly associated (p = 0.001) with a higher overall trial quality score. The categories of the lowest mean methodology scores were randomization and concealment, eligibility criteria, and reasons for patient exclusion, as identified with the Detsky score, and patient and assessor blinding, as identified with the risk-of-bias assessment.

CONCLUSIONS

The quantity and quality of published RCTs in JBJS have increased in the 2001 to 2013 time period compared with the previous time period. Although these improvements are encouraging, trends to smaller, single-center trials were also observed. To efficiently determine the efficacy of orthopaedic treatments and limit bias, high-quality randomized trials of appropriate sample size and rigorous design are needed.

Worldwide trends in volume and quality of published protocols of randomized controlled trials

Introduction

Publishing protocols of randomized controlled trials (RCT) facilitates a more detailed description of study rational, design, and related ethical and safety issues, which should promote transparency. Little is known about how the practice of publishing protocols developed over time. Therefore, this study describes the worldwide trends in volume and methodological quality of published RCT protocols.

Methods

A systematic search was performed in PubMed and EMBASE, identifying RCT protocols published over a decade from 1 September 2001. Data were extracted on quality characteristics of RCT protocols. The primary outcome, methodological quality, was assessed by individual methodological characteristics (adequate generation of allocation, concealment of allocation and intention-to-treat analysis). A comparison was made by publication period (First, September 2001- December 2004; Second, January 2005-May 2008; Third, June 2008-September 2011), geographical region and medical specialty.

Results

The number of published RCT protocols increased from 69 in the first, to 390 in the third period (p<0.0001). Internal medicine and paediatrics were the most common specialty topics. Whereas most published RCT protocols in the first period originated from North America (n = 30, 44%), in the second and third period this was Europe (respectively, n = 65, 47% and n = 190, 48%, p = 0.02). Quality of RCT protocols was higher in Europe and Australasia, compared to North America (OR = 0.63, CI = 0.40–0.99, p = 0.04). Adequate generation of allocation improved with time (44%, 58%, 67%, p = 0.001), as did concealment of allocation (38%, 53%, 55%, p = 0.03). Surgical protocols had the highest quality among the three specialty topics used in this study (OR = 1.94, CI = 1.09–3.45, p = 0.02).

Conclusion

Publishing RCT protocols has become popular, with a five-fold increase in the past decade. The quality of published RCT protocols also improved, although variation between geographical regions and across medical specialties was seen. This emphasizes the importance of international standards of comprehensive training in RCT methodology.

Characteristics and analysis of scientific articles submitted to the European Annals of Otorhinolaryngology, Head and Neck Diseases

OBJECTIVES

To evaluate characteristics, suggested modifications and reasons for rejection in scientific articles submitted for publication in the European Annals of Otorhinolaryngology, Head and Neck Diseases.

MATERIALS AND METHODS

A prospective study analyzed the flaws noted by reviewers in 52 scientific articles submitted to the European Annals of Otorhinolaryngology, Head and Neck Diseases between August 31, 2014 and February 28, 2015.

RESULTS

Fifteen flaws concerning content and 7 concerning form were identified. In more than 25% of submissions, major flaws were noted: purely descriptive paper; lack of contribution to existing state of knowledge; failure to define a clear study objective and/or analyze the impact of major variables; poorly structured Materials and methods section, lacking description of study population, objective and/or variables; lack of or inappropriate statistical analysis; Introduction verbose and/or misrepresenting the literature; excessively heterogeneous and/or poorly described study population; imprecise discussion, straying from the point, overstating the significance of results and/or introducing new results not mentioned in the Results section; description of the study population placed in the Results section instead of under Materials and methods; serious mistakes of syntax, spelling and/or tense; and failure to follow the Instructions to Authors. After review, 21.1% of articles were published, 65.3% rejected and 13.4% non-resubmitted within 3 months of review. On univariate analysis, the only variable increasing the percentage of articles accepted was the topic not being devoted to head and neck surgery (P=0.03).

CONCLUSION

These results document the excessive flaw rate still to be found in manuscripts and demonstrate the continuing need for authors to master and implement the rules of scientific medical writing.

Technical editing of research reports in biomedical journals

BACKGROUND

Most journals try to improve their articles by technical editing processes such as proof-reading, editing to conform to 'house styles', grammatical conventions and checking accuracy of cited references. Despite the considerable resources devoted to technical editing, we do not know whether it improves the accessibility of biomedical research findings or the utility of articles. This is an update of a Cochrane methodology review first published in 2003.

OBJECTIVES

To assess the effects of technical editing on research reports in peer-reviewed biomedical journals, and to assess the level of accuracy of references to these reports.

SEARCH STRATEGY

We searched The Cochrane Library Issue 2, 2007; MEDLINE (last searched July 2006); EMBASE (last searched June 2007) and checked relevant articles for further references. We also searched the Internet and contacted researchers and experts in the field.

SELECTION CRITERIA

Prospective or retrospective comparative studies of technical editing processes applied to original research articles in biomedical journals, as well as studies of reference accuracy.

DATA COLLECTION AND ANALYSIS

Two review authors independently assessed each study against the selection criteria and assessed the methodological quality of each study. One review author extracted the data, and the second review author repeated this.

MAIN RESULTS

We located 32 studies addressing technical editing and 66 surveys of reference accuracy. Only three of the studies were randomised controlled trials. A 'package' of largely unspecified editorial processes applied between acceptance and publication was associated with improved readability in two studies and improved reporting quality in another two studies, while another study showed mixed results after stricter editorial policies were introduced. More intensive editorial processes were associated with fewer errors in abstracts and references. Providing instructions to authors was associated with improved reporting of ethics requirements in one study and fewer errors in references in two studies, but no difference was seen in the quality of abstracts in one randomised controlled trial. Structuring generally improved the quality of abstracts, but increased their length. The reference accuracy studies showed a median citation error rate of 38% and a median quotation error rate of 20%.

AUTHORS' CONCLUSIONS

Surprisingly few studies have evaluated the effects of technical editing rigorously. However there is some evidence that the 'package' of technical editing used by biomedical journals does improve papers. A substantial number of references in biomedical articles are cited or quoted inaccurately.

Technical editing of research reports in biomedical journals

BACKGROUND

Most journals try to improve their articles by technical editing processes such as proof-reading, editing to conform to 'house styles' and grammatical conventions. Despite the considerable resources devoted to technical editing, we do not know whether it improves the accessibility of biomedical research findings or the utility of articles.

OBJECTIVES

To assess the effects of technical editing on research reports in peer-reviewed biomedical journals.

SEARCH STRATEGY

We searched the Cochrane Library Issue 1, 2001, MEDLINE (last searched February 2000), 12 other databases, handsearched 9 journals and checked relevant articles for further references. We also searched the Internet and contacted researchers and experts in the field.

SELECTION CRITERIA

Prospective or retrospective comparative studies of technical editing processes applied to original research articles in biomedical journals.

DATA COLLECTION AND ANALYSIS

Two reviewers independently assessed each study against the selection criteria and assessed the methodological quality of each study. One reviewer extracted the data, and the second reviewer repeated this.

MAIN RESULTS

We located 18 studies addressing technical editing and 35 surveys of reference accuracy. Only two of the studies were randomized controlled trials. A 'package' of largely unspecified editorial processes applied between acceptance and publication was associated with improved readability in two studies and improved reporting quality in another two studies, while another study showed mixed results after stricter editorial policies were introduced. More intensive editorial processes were associated with fewer errors in abstracts and references. Providing instructions to authors was associated with improved reporting of ethics requirements in one study and fewer errors in references in two studies, but no difference was seen in the quality of abstracts in one randomized controlled trial. Structuring generally improved the quality of abstracts, but increased their length. The reference accuracy studies showed a median citation error rate of 39% and a median quotation error rate of 20%.

AUTHORS' CONCLUSIONS

Surprisingly few studies have evaluated the effects of technical editing rigorously. However there is some evidence that the 'package' of technical editing used by biomedical journals does improve papers.

The reporting of randomized clinical trials using a surgical intervention is in need of immediate improvement: a systematic review

OBJECTIVE

To assess the reporting of surgical interventions, care providers, and number of centers in randomized clinical trials.

METHODS

Systematic review was performed to assess reports of randomized controlled trials assessing surgical procedure published in 2004. A standardized abstraction form was used to extract data.

RESULTS

A total of 158 articles were included. Details on the intervention intended, such as the surgical procedure, were reported in 138 (87.3%) articles, anesthetic management in 56 (35.4%), preoperative care in 34 (15.2%), and postoperative care in 78 (49.4%). How the experimental surgical intervention was carried out was reported in 64 articles (40.5%). Most trials were conducted in single centers (n = 109, 69.0%). The setting was reported in only 11 articles, and the volume of interventions performed was only reported in 5. Selection criteria were reported for care providers in 64 articles (40.5%). The number of care providers performing the intervention was reported in 51 articles (32.2%). The quality of reporting was low as assessed by CLEAR NPT (a 10-items checklist specifically developed to assess the reporting quality of RCTs assessing nonpharmacologic treatment).

CONCLUSIONS

Inadequate reporting on the management of the surgical procedure, care providers, and surgery center may introduce bias in RCTs of surgical interventions, making their results questionable. We recommend extending the CONSORT Statement to surgical interventions.

Evaluating the quality of active-control trials in periodontal research

AIM

The increasing popularity of randomized-controlled trials (RCTs) has raised the issue of their quality. Frequently overlooked are the differences between superiority and equivalence trials. The purpose of this study was to apply specific methodological criteria to evaluate the quality of active-control trials using studies that compared guided tissue regeneration (GTR) with enamel matrix derivatives (EMD).

MATERIALS AND METHODS

Seven RCTs were identified in the literature. Standard methodological criteria and seven additional criteria for trials using active-control groups were used to evaluate the quality of the seven RCTs.

RESULTS

Two trials were considered as superiority trials. The remaining five provided no clear statement of their research aim. However, two claimed that EMD and GTR were equally effective, because their results failed to show a significant difference between EMD and GTR. Most trials did not meet the majority of the design criteria.

CONCLUSIONS

The general lack of compliance with quality criteria might place doubt on the value of these trials and may render any conclusions questionable. It is therefore important to distinguish clearly between superiority trials and equivalence trials, and to incorporate appropriate additional criteria in the design of future RCTs with active-control groups.

The Consolidated Standards of Reporting Trials (CONSORT): better presentation of surgical trials in the Journal of Hand Surgery

To assure readers that study results are scientifically valid, the methods of a clinical trial should be described adequately. Since randomization, blinding, and intention-to-treat-analysis are major bias-reducing techniques, these aspects should be reported most accurately. The Consolidated standards of reporting trials (CONSORT) are recommendations to improve the reporting of trials. CONSORT requires that trial authors describe basic methodological aspects that readers need to appraise the strengths of reported clinical trials. This article presents the CONSORT recommendations and explains some of their main aspects. From now on, the Journal of Hand Surgery will use CONSORT to assist authors of randomized controlled trials in improving the description of their studies. We believe that this decision increases the scientific validity of study reports and helps readers when critically appraising articles.

The quality of reporting and outcome measures in randomized clinical trials related to upper-extremity disorders

PURPOSE

Randomized clinical trials can provide strong evidence regarding effective treatment options. The quality of reporting and the type of outcome measures used are important when judging whether results justify change in clinical practice. The aim of this study was to assess the quality of reporting of randomized clinical trials related to treatment of upper-extremity disorders, published in 4 hand surgical and orthopedic journals during an 11-year period, and assess the type of outcome measures used in the trials.

METHODS

Eligible articles were identified by reviewing all abstracts published in the 4 journals from 1992 through 2002. The quality of reporting was assessed by a modified Jadad scale that consisted of 3 items (randomization, blinding, and withdrawals/dropouts). A higher score (0-5) indicated higher quality. The outcome measures were classified according to the International Classification of Functioning, Disability and Health into the levels of body function and structure, activity, and participation.

RESULTS

Of 92 articles reporting randomized clinical trials, 40 articles described appropriate randomization method that implied they were truly randomized studies, 31 articles did not describe the randomization method, and 21 articles (23%) described inappropriate randomization methods. Double or single blinding was reported in 33 articles. Absence or description of withdrawals/dropouts was shown in 77 articles. The median quality score calculated for all 92 articles was 2 (range, 0-5) points. The median score for the 28 articles published 1992 through 1996 was 1 (range, 0-5) points and for the 64 articles published from 1997 through 2002 was 3 (range, 0-5) points. All trials used outcome measures on body function and structure level; 41% used measures of activity and/or participation.

CONCLUSIONS

There is a need to improve the quality of reporting of upper-extremity randomized clinical trials and to increase the use of outcome measures covering different aspects of disability.

Randomized Trials of Neurosurgical Interventions: A Systematic Appraisal

OBJECTIVE

To systematically appraise the study design and quality of reporting of randomized controlled trials (RCTs) on neurosurgical procedures and to identify potential defects and biases.

METHODS

Randomized controlled trials with at least five patients comparing any neurosurgical procedure against another procedure, nonsurgical treatment, or no treatment were retrieved from MEDLINE, EMBASE, and the Cochrane Library. We analyzed study design, quality of reporting, and trial results.

RESULTS

The median sample size in the 108 eligible reports was 68 patients. Ninety-nine trials (91.7%) reported inclusion and exclusion criteria, 55 (50.9%) mentioned the randomization mode, and 87 (80.6%) adequately described withdrawals, but only 31 (28.7%) described allocation concealment, only 23 (21.3%) gave power calculations, and only 20 (18.5%) were adequately powered. Significant efficacy or trend for efficacy was claimed in 46 reports (42.6%), and no difference between the compared procedures was found in 60 trials (55.6%). Trials with a larger sample size were more likely to report withdrawals (P = 0.02) and power calculations (P = 0.006). Only 14 trials (13.6%) were double-blind, and this was less frequent in longer trials (P = 0.02). Among quality criteria, only the reporting of randomization mode improved significantly over time (P = 0.015).

CONCLUSION

Several aspects of the design and reporting of randomized controlled trials on neurosurgical procedures can be improved. Larger, adequately powered, and accurately reported trials are needed.

Critical analysis of clinical research reporting in pediatric surgery: justifying the need for a new standard

BACKGROUND/PURPOSE

Clinical practice in surgery relies heavily on observational data in which accurate and nonbiased reporting is critical. This study aims to assess the adequacy of clinical research reporting in pediatric surgery and to develop a means to raise the standard of such reporting.

METHODS

The authors analyzed all observational studies published in The Journal of Pediatric Surgery from 1997 to 2002 (n = 300). Studies were assessed for 16 baseline criteria essential for the nonbiased reporting of clinical data (details regarding surgeons, cases, interventions, and statistical methods). Seven additional criteria pertaining to comparison methods were assessed in studies using controls.

RESULTS

Ninety-five percent of all studies were retrospective, and only 25% utilized a control group. Most studies met less than half of the essential reporting criteria (mean, 7.6 of 16 baseline criteria; 3.3 of 7 comparison criteria). Reporting deficiencies were found in all major aspects of study design and statistical analysis.

CONCLUSIONS

More rigorous reporting of clinical data in pediatric surgery could increase the clinical utility of published results. The authors have identified the fundamental elements essential to nonbiased reporting of clinical research data in surgery. Implementation of mandatory peer-review guidelines based on these principles could set a new standard for clinical reporting in surgery.

Orthopaedic fractures: trends in randomised controlled trials

AIM

To ascertain and describe the number and epidemiology of randomised controlled trials (RCTs) focused on orthopaedic fractures.

METHODS

A sensitive literature search was carried out for the period 1966-May 1999. Labels were applied to each identified RCT to indicate the fracture type, and the main type of intervention.

RESULTS

648 RCTs related to surgery of which 123 focused on adjuvant therapies and 88 related mainly to anaesthesia, analgesia, and radiography. The number of trials have increased exponentially with time so that the present decade has seen more RCTs published than all the other years added together.

CONCLUSION

There is clearly an encouraging trend in the number of RCTs published. However there is a need to ensure that trials are on fracture types where there is most need for guidance. This growing evidence base should fuel systematic reviews and clinical guidelines within orthopaedics.

Methodology standards associated with quality reporting in clinical studies in pediatric surgery journals

BACKGROUND/PURPOSE

Reports of clinical trials often lack adequate descriptions of design and analysis; recent attention has focused on improving this omission so readers can properly assess the strength of the findings and draw their own conclusions. Similar analysis of study design and methodologic standards associated with quality reporting has not been carried out for pediatric surgery journals.

METHODS

All studies (n = 642) published in 1998 in Journal of Pediatric Surgery (JPS) and Pediatric Surgery International (PSI), were reviewed for demographic data and study design. The frequency of reporting of 11 basic elements of design and analysis was evaluated in randomized clinical trials (RCT), nonrandomized clinical trials (NRCT), and retrospective cohorts (RC) from JPS by consensus of 2 assessors.

RESULTS

Of the 642 studies, 17% of articles (111 of 642) were classified as clinical studies. Sixty-three were comparative studies and consisted of RC (n = 48), NRCT (n = 12), and RCT (n = 3). Two-thirds of articles published were either case reports or case series (431 of 642), and 16% were basic science articles. Demographic analysis showed a wide range of topics addressed, 4 authors per article, and multiple country of origin of authors. More than 66% of all RCT in JPS reported on eligibility criteria, admission before allocation, random allocation, method of randomization, patients' blindness to treatment, treatment complications, statistical analyses, statistical methods, loss to follow-up, and statistical methods; 2 elements of design and analysis, however, were poorly reported: blind assessment of outcome (33%) and power (17%).

CONCLUSIONS

There were few randomized, controlled trials in pediatric surgery journals, and further attention should be given to evaluate the causal factors. Nine elements of quality reporting were well reported; however, 2 others were poorly reported; this may improve if editors of pediatric surgical journals provide authors with guidelines on how to report clinical trial design and analysis.