Methodological reporting quality of randomized controlled trials: A survey of seven core journals of orthopaedics from Mainland China over 5 years following the CONSORT statement

Patient reported outcome measures (PROMs) as primary and secondary outcomes in total hip and knee arthroplasty randomized controlled trials: a systematic review

BACKGROUND

Significant heterogeneity exists regarding patient reported outcome measures (PROMs) used in total hip (THA) and knee (TKA) arthroplasty randomized controlled trials (RCTs). This study investigates the PROMs used as primary and secondary outcomes in contemporary arthroplasty RCTs.

METHODS

A literature search identified THA and TKA RCTs that were published in top ten impact factor orthopaedic journals from 2017 to 2021. Screening identified 241 trials: 76 THA, 157 TKA, and eight combined. Data were extracted to identify PROMs utilized as either primary or secondary outcomes and the time period of measurement.

RESULTS

Visual Analog Scale (VAS) Pain was the most reported primary PROM in THA (9.2%) and TKA (22.9%) trials. This was followed by Numeric Rating Scale (NRS) Pain (7.9%) and the Harris Hip score (6.6%) in THA trials and NRS Pain (4.5%) and the Knee Society score (4.5%) in TKA trials. Many THA (37.0%) and TKA (52.1%) trials did not clearly specify primary outcome time points. Only pain scales were reported at time points less than one week, while various joint-specific functional outcomes were reported at later time points. As secondary outcomes, the Harris Hip score (28.9%) was most common in THA trials and the Knee Society score (26.1%) was most common in TKA trials. Indeterminate primary or secondary outcomes were reported in 18.2% of studies.

CONCLUSIONS

Contemporary THA and TKA trials exhibit heterogeneity of PROMs as study outcomes after the first postoperative week. Our findings highlight the need for consensus in PROM reporting and better methodological reporting to improve the interpretability of RCT outcomes.

PROSPERO REGISTRATION NUMBER

CRD42022337255.

Spin in the abstracts of randomized controlled trials of nurse-led care: A cross-sectional study

BACKGROUND

Nurse-led models of care had been reported as effective intervention approaches for improving health management and reducing hospitalizations of target patients in a number of studies. However, the reporting quality of studies in the field varied and there was a lack of literature evaluation.

OBJECTIVE

The objective of the study was to assess the reporting quality and spin of abstracts of published randomized controlled trials which had statistically not significant primary outcomes. Moreover, potential factors associated with the presence of spin were also assessed.

METHODS

Studies on nurse-led care were retrieved from PubMed from January 1st, 2017, to December 31st, 2021. Only randomized controlled trials with statistically not significant primary outcomes were included. Study screening and data extraction were carried out by two reviewers independently. The reporting quality of each abstract was evaluated by the Consolidated Standards of Reporting Trials statement, and spin strategies were analyzed using a pre-designed assessment form. Potential predictors for the presence of spin were analyzed by multivariate logistic regressions.

RESULTS

The overall reporting quality of the included 75 randomized controlled trial abstracts was not satisfying, with a median score of 16-item Consolidated Standards of Reporting Trials statement at 6 (IQR 5, 8). Forty abstracts used at least one spin strategy in abstracts. Among them, 18 (45.0 %) used spin strategies in the result section and 39 (97.5 %) had spin in the conclusion section. The most common spin strategy identified in abstracts was focusing on statistically significant secondary outcomes (12/40, 30.0 %) in the result section and claiming benefit with no consideration of statistically not significant results for the primary outcomes (32/40, 80.0 %) in the conclusion section. Based on the definition, 29 (72.5 %) abstracts were assessed to have high level of spin in the conclusions of abstracts. By multivariate logistic regression analyses, it was found that only geographic origin (reference: studies from Asian countries, OR = 0.118, 95 % CI 0.027 to 0.511, P = 0.004) and the Consolidated Standards of Reporting Trials statement score (reference: lower score, OR = 0.625, 95 % CI 0.470 to 0.829, P = 0.001) were significantly associated with the presence of spin in abstracts.

CONCLUSION

Among the randomized controlled trials with statistically not significant primary outcomes in the field of nurse-led care, the reporting quality of abstracts needs to be improved. Trials from Asian countries and with lower Consolidated Standards of Reporting Trials statement scores are more likely to present spin in abstracts. Findings reported in the result and conclusion sections of these abstracts need to be interpreted with caution.

Assessment of 30 Years of Randomized Controlled Trials in The American Journal of Sports Medicine: 1990-2020

Background

Randomized controlled trials (RCTs) stand atop the evidence-based hierarchy of study designs for their ability to arrive at results with the lowest risk of bias. Even for RCTs, however, critical appraisal is essential before applying results to clinical practice.

Purpose

To analyze the quality of reporting of RCTs published in The American Journal of Sports Medicine (AJSM) from 1990 to 2020 and to identify trends over time and areas of improvement for future trials.

Study Design

Systematic review; Level of evidence, 1.

Methods

We queried the AJSM database for RCTs published between January 1990 and December 2020. Data pertaining to study characteristics were recorded. Quality assessments were conducted using the Detsky quality-of-reporting index and the modified Cochrane risk-of-bias (mROB) tool. Univariate and multivariable models were generated to establish factors with associations to study quality. The Fragility Index was calculated for eligible studies.

Results

A total of 277 RCTs were identified with a median sample size of 70 patients. A total of 19 RCTs were published between 1990 and 2000 (t₁); 82 RCTs between 2001 and 2010 (t₂); and 176 RCTs between 2011 and 2020 (t₃). From t₁ to t₃, significant increases were observed in the overall mean-transformed Detsky score (from 68.2% ± 9.8% to 87.4% ± 10.2%, respectively; P < .001) and mROB score (from 4.7 ± 1.6 to 6.9 ± 1.6, respectively; P < .001). Multivariable regression analysis revealed that trials with follow-up periods of <5 years clearly stated primary outcomes, and a focus on the elbow, shoulder, or knee were associated with higher mean-transformed Detsky and mROB scores. The median Fragility Index was 2 (interquartile range, 0-5) for trials with statistically significant. Studies with small sample sizes (<100 patients) were more likely to have low Fragility Index scores and less likely to have a statistically significant finding in any outcome.

Conclusion

The quantity and quality of published RCTs published in AJSM increased over the past 3 decades. However, single-center trials with small sample sizes were prone to fragile results.

What are the rates and validity of French authors' conflicts of interest disclosures in Orthopaedics & Traumatology: Surgery & Research? Analysis of the year 2017 with comparison versus the Transparence.sante.gouv.fr database

INTRODUCTION

Authors are responsible for their own Disclosure of Interests in submitting to Orthopaedics & Traumatology: Surgery & Research (OTSR). Disclosure may be incomplete or false, by deliberate or unintended omission. Omission may impact the analysis of results or at least the reader's perception of it. The rate of validity of disclosure in OTSR is not known, and we therefore conducted a bibliometric study on articles published in 2017, in order to: 1) determine the proportion of articles in which authors disclose interests; and 2), when interests are disclosed, to assess their validity by comparison against the Transparence.gov.fr database.

HYPOTHESIS

Disclosure rates exceed 75%.

MATERIAL AND METHOD

We analyzed all articles published in OTSR in 2017, and extracted those with at least one French author. We also analyzed data from the Transparence.gouv.fr corporate declaration database, for comparison with the authors' own disclosures in each article. Significant interest was defined by a €1000 threshold, although higher thresholds (>€5000, >€10,000 and >€100,000) were also investigated.

RESULTS

In 2017, 337 articles were published in OTSR, 210 of which had at least 1 French author. Of these, 201 (95.7%) had at least 1 author with significant interest (>€1000) according to the Transparence data. In 189 of these 201 articles (94%), authors had failed to disclose at least 1 direct or indirect conflict of interest. This omission rate fell to 22/45 (48.9%) for more substantial interests, which were more faithfully disclosed. At the €1000 threshold, in only 8/201 articles (4%) did authors disclose all their interests with perfect validity, but this rate increased up to 25/45 (55.5%) at higher thresholds. At the €1000 threshold, 66 of the 201 articles cited the trade-name of interest, resulting in 54/66 (82%) correct disclosure of direct interest; this rate increased up to 25/26 (96%) at higher thresholds.

DISCUSSION

At a threshold of €1000, the rate of complete and valid disclosure was 4% and the rate of omission 94%. At higher thresholds, rates were better, with just 48.9% omission and, above all, 55.5% validity, even if these rates were lower than hypothesized (75%). Authors and editors need to take greater care. Disclosures were often made, but incompletely, and authors need reminding that they must disclose not only interests related to the article in question but also all interests that might directly or indirectly influence their interpretation of the results reported, allowing readers to make up their own minds.

LEVEL OF EVIDENCE

IV; systematic retrospective study.

The methodological quality on systematic reviews of surgical randomised controlled trials: A cross-sectional survey

BACKGROUND

Systematic reviews of RCTs have been developed to address end-users' needs and are regarded as the highest level of evidence. Flaws in the design, conduct and analyses of a systematic review can lead to erroneous conclusions and increase the research waste.

OBJECTIVE

We undertook a cross-sectional survey to identify the critical areas of weakness in systematic reviews for surgical interventions by AMSTAR 2.

METHODS

We searched PubMed, EMbase and Cochrane Library to summarize systematic reviews of surgical RCTs published in 2017. The information regarding general characteristics and methodological characteristics were gathered. We conducted descriptive analyses of study characteristics of included systematic reviews and explored the difference among varied methodological quality.

RESULTS

Totally 141 systematic reviews were identified. We found only four reviews (2.8%) were high quality, 3 (2.1%) were moderate quality, 8 (5.7%) were low quality, and the remaining 126 (89.4%) were of critical low quality. The critical weaknesses were lack of pre-registration or published protocols (29.1%), comprehensive literature search (17.7%), lists of excluded studies and reasons for exclusion (19.1%), description of detailed interventions (8.5%), extraction of funding source from included trials (10.6%), and consideration of the risk of bias of included trials when synthesized (16.3%) and interpret (20.6%) the results. Higher methodological quality was only positively associated with Cochrane systematic review.

CONCLUSION

Although two-thirds of included systematic reviews in the field of surgery were published in journals ranking Q1, the methodological quality is suboptimal and needs to be substantially improved. More efforts of multi-disciplinary teams' collaboration, continual education and training, integrally connection between primary studies and systematic review and contributing surgical research to practice should be imperative.

Reporting of statistical sample size calculations in publications of trials on age-related macular degeneration, glaucoma and cataract

Background

Transparent and complete publications of randomised controlled trials (RCT) ought to comply with the guidelines of the CONSORT Statement, which stipulates sample size calculation as an important aspect of trial planning. The objective of this study was to analyse and compare the reporting of statistical sample size calculations in RCT papers on the treatment of age-related macular degeneration (AMD), glaucoma and cataract published in 2018.

Material and methods

This study comprises a total of 113 RCT papers (RCT-P) published in 2018 (AMD: 14, glaucoma: 28, cataract: 71), in English or German, and identified through an internet-based literature search in PubMed and EMBASE. The primary outcome measure of the study was the number of trials providing a complete description of the underlying sample case calculation on the basis of the variables required (significance level, expected outcomes, power, and resulting sample size).

Results

Of the RCTs reviewed, 64% (AMD), 61% (glaucoma) and 31% (cataract) provided a justification of the number of patients included. A complete description of the described studies’ sample size calculation including all the necessary values (primary outcome measure of this study) was described by 21% of the AMD, 29% of the cataract and 18% of the glaucoma RCT publications (in total: 24 of 113 (21%) at a confidence interval of 95%: [13%; 29%]).

Conclusion

All three treatment areas analysed lacked reporting quality regarding the justification of the number of patients included in a clinical trial based on a sample size calculation required for ethical reasons. More than half of all RCT publications reviewed did not provide all of the required information on statistical sample size calculation, and thus lacked transparency and completeness. It is therefore urgently required to involve methodologists in a study’s planning and publishing processes to ensure that methodology descriptions are transparent and of high quality.

Chinese medicine formulas for nonalcoholic fatty liver disease: Overview of systematic reviews

BACKGROUND

Nonalcoholic fatty liver disease (NAFLD) affects more than one-quarter of the global population. Due to the lack of approved chemical agents, many patients seek treatment from traditional Chinese medicine (TCM) formulas. A variety of systematic reviews have been published regarding the effectiveness and safety of TCM formulas for NAFLD.

AIM

To critically appraise available systematic reviews and sort out the high-quality evidence on TCM formulas for the management of NAFLD.

METHODS

Seven databases were systematically searched from their inception to 28 February 2020. The search terms included “non-alcoholic fatty liver disease,” “Chinese medicines,” “systematic review,” and their synonyms. Systematic reviews involving TCM formulas alone or in combination with conventional medications were included. The methodological quality and risk of bias of eligible systematic reviews were evaluated by using A Measure Tool to Assess Systematic Reviews 2 (AMSTAR 2) and Risk of Bias in Systematic Review (ROBIS). The quality of outcomes was assessed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system.

RESULTS

Seven systematic reviews were ultimately included. All systematic reviews were conducted based on randomized controlled trials and published in the last decade. According to the AMSTAR 2 tool, one systematic review was judged as having a moderate confidence level, whereas the other studies were rated as having a low or extremely low level of confidence. The ROBIS tool showed that the included systematic reviews all had a high risk of bias due to insufficient consideration of identified concerns. According to the GRADE system, only two outcomes were determined as high quality; namely, TCM formulas with the HuoXueHuaYu principle were better than conventional medications in ultrasound improvement, and TCM formulas were superior to antioxidants in alanine aminotransferase normalization. Other outcomes were downgraded to lower levels, mainly because of heterogeneity among studies, not meeting optimal information sample size, and inclusion of excessive numbers of small sample studies. Nevertheless, the evidence quality of extracted outcomes should be further downgraded when applying to clinical practice due to indirectness.

CONCLUSION

The quality of available systematic reviews was not satisfactory. Researchers should avoid repeatedly conducting systematic reviews in this area and focus on designing rigorous randomized controlled trials to support TCM formula applications.

A systematic survey showed important limitations in the methods for assessing drug safety among systematic reviews

OBJECTIVES

This study aimed to examine the design, conduct, and analysis of systematic reviews assessing drug safety through a cross-sectional survey.

STUDY DESIGN AND SETTING

We searched PubMed to identity systematic reviews published in the Cochrane Database of Systematic Reviews and Core Clinical Journals indexed in 2015 and randomly sampled systematic reviews assessing drug effects at a 1:1 ratio of Cochrane and non-Cochrane reviews. Teams of two investigators independently conducted study screening and collected data, using prespecified, standardized questionnaires. In addition to general information, we collected details about the planning and analyses of safety outcomes.

RESULTS

We included 120 systematic reviews, including 60 Cochrane and 60 non-Cochrane reviews. Most reviews searched PubMed/MEDLINE (n = 117, 97.5%), EMBASE (n = 105, 87.5%), and Cochrane CENTRAL (n = 110, 91.7%) and conducted independent and duplicate study selection (n = 98, 81.7%), risk of bias assessment (n = 105, 87.5%), and data collection (n = 105, 87.5%). Only nine (7.5%) reviews clearly defined safety outcomes, and seven (5.8%) defined a primary safety outcome; none stated whether the primary safety outcome was predefined. Among the 80 reviews that pooled the primary dichotomous safety data across studies, less than half (41%, n = 33) conducted subgroup analysis to explore for sources of heterogeneity or reported a GRADE assessment for the overall quality of evidence. Cochrane reviews were more likely to provide a study protocol (100% vs. 23.3%; P < 0.001), involve methodologists (53.3% vs. 20.0%; P < 0.001), and report a GRADE assessment for the primary safety outcome (70.6% vs. 19.6%; P < 0.001).

CONCLUSION

Our findings highlighted areas for improved planning and analysis in the assessment of drug safety among systematic reviews. Cochrane reviews were superior to non-Cochrane reviews; however, most reviews did not prespecify their safety outcomes or methods for analysis, explore sources of heterogeneity among pooled effects, or assess the overall quality of evidence with the GRADE approach.

Quality assessment of controlled clinical trials published in Orthopaedics and Traumatology journals in Spanish: An observational study through handsearching and evidence mapping

Few Orthopaedics and Traumatology journals from Latin America and Spain are indexed in major databases; controlled clinical trials published in these journals cannot be exhaustively retrieved using electronic literature searches. We aimed to identify, describe and assess the quality of controlled clinical trials published in Orthopaedics and Traumatology journals from Latin America and Spain through handsearching and evidence mapping methods. We identified controlled clinical trials published in eligible Orthopaedics/Traumatology journals in Spanish until July 2017 by handsearching. Data were extracted for controlled clinical trials main characteristics and the Cochrane risk of bias tool was used to assess the controlled clinical trials methodological quality. In addition, we mapped the main findings of these trials. As a result, we assessed 5631 references in 29 eligible journals of which 57 were controlled clinical trials (1.0%). Controlled clinical trials were published between 1995 and 2017 at a rate of 2.5 per year. Journals from Spain and Mexico published around 63% of the controlled clinical trials identified. The median sample size of patients enrolled was 60 (range = 30-300 participants). About conditions assessed, 38.5% of controlled clinical trials assessed issues related to knee conditions, 15.7% about hip and 10.5% about trauma or spine. The risk of bias domains most affected was selective reporting bias and random sequence generation. In addition, only two and seven trials had low risk of bias in all items related to participant/personnel and outcome assessment blindings, respectively. More than 40% of studies did not report differences on benefits/harms between the interventions assessed. As a conclusion, the number of controlled clinical trials published in Orthopaedics/Traumatology journals from Latin America and Spain is low. These controlled clinical trials had important methodological shortcomings and were judged as unclear or high risk of bias. These trials are now available in CENTRAL for their potential inclusion in systematic reviews and other documents of synthesis.

The study design elements employed by researchers in preclinical animal experiments from two research domains and implications for automation of systematic reviews

Systematic reviews are increasingly using data from preclinical animal experiments in evidence networks. Further, there are ever-increasing efforts to automate aspects of the systematic review process. When assessing systematic bias and unit-of-analysis errors in preclinical experiments, it is critical to understand the study design elements employed by investigators. Such information can also inform prioritization of automation efforts that allow the identification of the most common issues. The aim of this study was to identify the design elements used by investigators in preclinical research in order to inform unique aspects of assessment of bias and error in preclinical research. Using 100 preclinical experiments each related to brain trauma and toxicology, we assessed design elements described by the investigators. We evaluated Methods and Materials sections of reports for descriptions of the following design elements: 1) use of comparison group, 2) unit of allocation of the interventions to study units, 3) arrangement of factors, 4) method of factor allocation to study units, 5) concealment of the factors during allocation and outcome assessment, 6) independence of study units, and 7) nature of factors. Many investigators reported using design elements that suggested the potential for unit-of-analysis errors, i.e., descriptions of repeated measurements of the outcome (94/200) and descriptions of potential for pseudo-replication (99/200). Use of complex factor arrangements was common, with 112 experiments using some form of factorial design (complete, incomplete or split-plot-like). In the toxicology dataset, 20 of the 100 experiments appeared to use a split-plot-like design, although no investigators used this term. The common use of repeated measures and factorial designs means understanding bias and error in preclinical experimental design might require greater expertise than simple parallel designs. Similarly, use of complex factor arrangements creates novel challenges for accurate automation of data extraction and bias and error assessment in preclinical experiments.

Critical components for designing and implementing randomized controlled trials

Randomized controlled trials (RCTs) are considered the first level of evidence to assess the efficacy of novel interventions/therapies. Proper design and implementation of an RCT can result in convincing causal inferences. RCTs often represent the gold standard for clinical trials when appropriately designed, conducted and reported. However, there are limitations in implementation of RCTs, including sufficiency of randomized allocation (especial for allocation concealment), implementing standard intervention, maintaining follow-up and statement of conflicting interests. Therefore, the basic principles of RCTs are outlined here so that pediatric investigators can further understand what is the best evidence based on RCTs. More importantly, the quality of pediatric RCTs may be improved by following challenges in pediatric clinical trials outlined here.