Interpretation of CIs in clinical trials with non-significant results: systematic review and recommendations

Evidence of Lack of Treatment Efficacy Derived From Statistically Nonsignificant Results of Randomized Clinical Trials

Importance

Many randomized clinical trials yield statistically nonsignificant results. Such results are difficult to interpret within the dominant statistical framework.

Objective

To estimate the strength of evidence in favor of the null hypothesis of no effect vs the prespecified effectiveness hypothesis among nonsignificant primary outcome results of randomized clinical trials by application of the likelihood ratio.

Design, Setting, and Participants

Cross-sectional study of statistically nonsignificant results for primary outcomes of randomized clinical trials published in 6 leading general medical journals in 2021.

Outcome measures

The likelihood ratio for the null hypothesis of no effect vs the effectiveness hypothesis stated in the trial protocol (alternate hypothesis). The likelihood ratio quantifies the support that the data provide to one hypothesis vs the other.

Results

In 130 articles that reported 169 statistically nonsignificant results for primary outcomes, 15 results (8.9%) favored the alternate hypothesis (likelihood ratio, <1), and 154 (91.1%) favored the null hypothesis of no effect (likelihood ratio, >1). For 117 (69.2%), the likelihood ratio exceeded 10; for 88 (52.1%), it exceeded 100; and for 50 (29.6%), it exceeded 1000. Likelihood ratios were only weakly correlated with P values (Spearman r, 0.16; P = .045).

Conclusions

A large proportion of statistically nonsignificant primary outcome results of randomized clinical trials provided strong support for the hypothesis of no effect vs the alternate hypothesis of clinical efficacy stated a priori. Reporting the likelihood ratio may improve the interpretation of clinical trials, particularly when observed differences in the primary outcome are statistically nonsignificant.

Cognitive behavioural group therapy as addition to psychoeducation and pharmacological treatment for adolescents with ADHD symptoms and related impairments: a randomised controlled trial

BACKGROUND

Cognitive behavioural therapy (CBT) is recommended for attention-deficit/hyperactivity-disorder (ADHD) in adolescents. However, all CBTs are not created equal, and the guidelines do not specify which CBT interventions are the most effective for this patient group. This study examines the efficacy of a group CBT without parent involvement as follow-up treatment compared to no additional CBT in adolescents with persistent and impairing ADHD symptoms after a short psychoeducational intervention and medical treatment.

METHODS

The authors conducted a two-arm parallel randomized controlled trial in two child and adolescent mental health outpatient clinics in Norway. One hundred patients aged 14-18 years with a diagnosis of ADHD (66%) or subthreshold ADHD (34%) were randomized to either a 12-week group CBT program (N = 50) or a non-CBT control condition (N = 50). Assessments were made at admission to the clinic, two weeks before and two weeks after treatment. The primary outcomes were parent-, teacher- and self-ratings of ADHD symptoms (ADHD Rating Scale-IV), and the secondary outcomes were ratings of ADHD symptom severity, executive function, functional impairment, and emotional problems. Evaluators blinded to group allocation rated ADHD symptom severity with the Clinical Global Impression Scale for Severity (CGI-S) at baseline and post-treatment.

RESULTS

Analyses using mixed-effects models showed no difference between the treatment arms from baseline to post treatment in primary and secondary outcomes.

CONCLUSIONS

Contrary to our hypothesis, we found no incremental treatment effect on the part of a group CBT as follow-up to psychoeducation and pharmacological treatment on ADHD symptoms and accompanying impairments. Limitations with the CBT was the large number and low dosage of treatment components, causing restricted time for practice. Unlike evidence-based, individualized targeted CBTs with parent involvement, a group CBT directed solely at the adolescents with no parent involvement does not appear effective for treating ADHD.

TRIAL REGISTRATION

NCT02937142 , 18/10/2016.

Evaluation of the psychometric properties of patient-reported and clinician-reported outcome measures of chemotherapy-induced peripheral neuropathy: a COSMIN systematic review protocol

INTRODUCTION

Chemotherapy-induced peripheral neuropathy (CIPN) is a poorly understood side effect of many antineoplastic agents. Patients may experience sensory, motor and autonomic symptoms, negatively impacting quality of life. A gold-standard assessment methodology has yet to be determined, limiting efforts to identify effective agents to prevent or treat CIPN.

METHODS AND ANALYSIS

This is a protocol of a systematic review of psychometric analyses of CIPN Clinician Reported Outcome Measures (ClinROM) and Patient-Reported Outcome Measures (PROM) among adults receiving, or who had previously received chemotherapy for cancer. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) quality ratings will be compared across studies and across ClinROMs and PROMs. Studies reporting psychometric proprieties of CIPN ClinROMs and/or PROMs among adults aged ≥18 years will be eligible for inclusion, with no restriction on language or year of publication. MEDLINE, Embase, CINAHL and APA PsycINFO databases will be searched from inception to 31 December 2021. Study characteristics, measurement properties of the ClinROMs and/or PROMs and the CIPN definitions will be extracted. The Synthesis Without Meta-analysis guideline will be used to guide data synthesis. The COSMIN Risk of Bias checklist will be used by two independent raters to assess methodological quality. Subgroup analyses by age, chemotherapy type, and study timing in relation to the delivery of chemotherapy will be carried out where data are available. An adapted version of Outcome Measures in Rheumatology filter 2.1 will be used to provide a best-evidence synthesis of CIPN ClinROMs and PROMs and to recommend a CIPN assessment tool for clinical and research settings.

ETHICS AND DISSEMINATION

Ethical approval is not necessary to be obtained for this systematic review protocol. Results will be disseminated to clinicians and policy-makers by publication in a peer-reviewed journal and by presenting at relevant conferences.

PROSPERO REGISTRATION NUMBER

CRD42021278168.

An implantable restorative-neurostimulator for refractory mechanical chronic low back pain: a randomized sham-controlled clinical trial

ABSTRACT

Chronic low back pain can be caused by impaired control and degeneration of the multifidus muscles and consequent functional instability of the lumbar spine. Available treatment options have limited effectiveness and prognosis is unfavorable. We conducted an international randomized, double-blind, sham-controlled trial at 26 multidisciplinary centers to determine safety and efficacy of an implantable, restorative neurostimulator designed to restore multifidus neuromuscular control and facilitate relief of symptoms (clinicaltrials.gov identifier: NCT02577354). Two hundred four eligible participants with refractory mechanical (musculoskeletal) chronic LBP and a positive prone instability test indicating impaired multifidus control were implanted and randomized to therapeutic (N = 102) or low-level sham (N = 102) stimulation of the medial branch of the dorsal ramus nerve (multifidus nerve supply) for 30 minutes twice daily. The primary endpoint was the comparison of responder proportions (≥30% relief on the LBP visual analogue scale without analgesics increase) at 120 days. After the primary endpoint assessment, participants in the sham-control group switched to therapeutic stimulation and the combined cohort was assessed through 1 year for long-term outcomes and adverse events. The primary endpoint was inconclusive in terms of treatment superiority (57.1% vs 46.6%; difference: 10.4%; 95% confidence interval, -3.3% to 24.1%, P = 0.138). Prespecified secondary outcomes and analyses were consistent with a modest but clinically meaningful treatment benefit at 120 days. Improvements from baseline, which continued to accrue in all outcome measures after conclusion of the double-blind phase, were clinically important at 1 year. The incidence of serious procedure- or device-related adverse events (3.9%) compared favorably with other neuromodulation therapies for chronic pain.

Current use of effect size or confidence interval analyses in clinical and biomedical research

The isolated use of the statistical hypothesis testing for two group comparison has limitations, and its combination with effect size or confidence interval analysis as complementary statistical tests is recommended. In the present work, we estimate the use of these complementary statistical tests (i.e. effect size or confidence interval) in recently published in research articles in clinical and biomedical areas. Methods: The ProQuest database was used to search published studies in academic journals between 2019 and 2020. The analysis was carried out using terms that represent five areas of clinical and biomedical research: "brain", "liver", "heart", "dental", and "covid-19". A total of 119,558 published articles were retrieved. Results: The relative use of complementary statistical tests in clinical and biomedical publications was low. The highest frequency usage of complementary statistical tests was among articles that also used statistical hypothesis testing for two-sample comparison. Publications with the term "covid-19" showed the lowest usage rate of complementary statistical tests when all article were analyzed but presented the highest rate among articles that used hypothesis testing. Conclusion: The low use of effect size or confidence interval in two-sample comparison suggests that coordinate measures should be taken in order to increase the use of this analysis in clinical and biomedical research. Their use should be emphasized in statistical disciplines for college and graduate students, become a routine procedure in research laboratories, and recommended by reviewers and editors of scientific journals.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11192-021-04150-3.

Effectiveness of a nurse-led hospital-to-home transitional care intervention for older adults with multimorbidity and depressive symptoms: A pragmatic randomized controlled trial

OBJECTIVE

To evaluate the effectiveness of a nurse-led hospital-to-home transitional care intervention versus usual care on mental functioning (primary outcome), physical functioning, depressive symptoms, anxiety, perceived social support, patient experience, and health service use costs in older adults with multimorbidity (≥ 2 comorbidities) and depressive symptoms.

DESIGN AND SETTING

Pragmatic multi-site randomized controlled trial conducted in three communities in Ontario, Canada. Participants were allocated into two groups of intervention and usual care (control).

PARTICIPANTS

127 older adults (≥ 65 years) discharged from hospital to the community with multimorbidity and depressive symptoms.

INTERVENTION

This evidence-based, patient-centred intervention consisted of individually tailored care delivery by a Registered Nurse comprising in-home visits, telephone follow-up and system navigation support over 6-months.

OUTCOME MEASURES

The primary outcome was the change in mental functioning, from baseline to 6-months. Secondary outcomes were the change in physical functioning, depressive symptoms, anxiety, perceived social support, patient experience, and health service use cost, from baseline to 6-months. Intention-to-treat analysis was performed using ANCOVA modeling.

RESULTS

Of 127 enrolled participants (63-intervention, 64-control), 85% had six or more chronic conditions. 28 participants were lost to follow-up, leaving 99 (47 -intervention, 52-control) participants for the complete case analysis. No significant group differences were seen for the baseline to six-month change in mental functioning or other secondary outcomes. Older adults in the intervention group reported receiving more information about health and social services (p = 0.03) compared with the usual care group.

CONCLUSIONS

Although no significant group differences were seen for the primary or secondary outcomes, the intervention resulted in improvements in one aspect of patient experience (information about health and social services). The study sample fell below the target sample (enrolled 127, targeted 216), which can account for the non-significant findings. Further research on the impact of the intervention and factors that contribute to the results is recommended.

TRIAL REGISTRATION

clinicaltrials.gov Identifier: NCT03157999.

Research design considerations for randomized controlled trials of spinal cord stimulation for pain: Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials/Institute of Neuromodulation/International Neuromodulation Society recommendations

ABSTRACT

Spinal cord stimulation (SCS) is an interventional nonpharmacologic treatment used for chronic pain and other indications. Methods for evaluating the safety and efficacy of SCS have evolved from uncontrolled and retrospective studies to prospective randomized controlled trials (RCTs). Although randomization overcomes certain types of bias, additional challenges to the validity of RCTs of SCS include blinding, choice of control groups, nonspecific effects of treatment variables (eg, paresthesia, device programming and recharging, psychological support, and rehabilitative techniques), and safety considerations. To address these challenges, 3 professional societies (Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials, Institute of Neuromodulation, and International Neuromodulation Society) convened a meeting to develop consensus recommendations on the design, conduct, analysis, and interpretation of RCTs of SCS for chronic pain. This article summarizes the results of this meeting. Highlights of our recommendations include disclosing all funding source and potential conflicts; incorporating mechanistic objectives when possible; avoiding noninferiority designs without internal demonstration of assay sensitivity; achieving and documenting double-blinding whenever possible; documenting investigator and site experience; keeping all information provided to patients balanced with respect to expectation of benefit; disclosing all information provided to patients, including verbal scripts; using placebo/sham controls when possible; capturing a complete set of outcome assessments; accounting for ancillary pharmacologic and nonpharmacologic treatments in a clear manner; providing a complete description of intended and actual programming interactions; making a prospective ascertainment of SCS-specific safety outcomes; training patients and researchers on appropriate expectations, outcome assessments, and other key aspects of study performance; and providing transparent and complete reporting of results according to applicable reporting guidelines.

Randomized Trial of Oral Iron and Diet Advice versus Diet Advice Alone in Young Children with Nonanemic Iron Deficiency

OBJECTIVE

To compare the effects of 2 treatment options on neurodevelopmental and laboratory outcomes in young children with nonanemic iron deficiency.

STUDY DESIGN

A blinded, placebo-controlled, randomized trial of children 1-3 years with nonanemic iron deficiency (hemoglobin ≥110 g/L, serum ferritin <14 μg/L) was conducted in 8 primary care practices in Toronto, Canada. Interventions included ferrous sulfate or placebo for 4 months; all parents received diet advice. The primary outcome was the Early Learning Composite (ELC) using the Mullen Scales of Early Learning (mean 100, SD 15). Secondary outcomes included serum ferritin. Measurements were obtained at baseline and 4 and 12 months. Sample size was calculated to detect a between-group difference of 6-7 points in ELC.

RESULTS

At enrollment (n = 60), mean age was 24.2 (SD 7.4) months and mean serum ferritin was 10.0 (SD 2.4) μg/L. For ELC, the mean between-group difference at 4 months was 1.1 (95% CI -4.2 to 6.5) and at 12 months was 4.1 (95% CI -1.9 to 10.1). For serum ferritin, at 4 months, the mean between-group difference was 16.9 μg/L (95% CI 6.5 to 27.2), and no child randomized to ferrous sulfate had a serum ferritin <14 μg/L (0% vs 31%, P = .003).

CONCLUSIONS

For young children with nonanemic iron deficiency, treatment options include oral iron and/or diet advice. We remain uncertain about which option is superior with respect to cognitive outcomes; however, adding ferrous sulfate to diet advice resulted in superior serum ferritin outcomes after 4 months. Shared decision-making between practitioners and parents may be considered when selecting either option.

TRIAL REGISTRATION

Clinicaltrials.gov: NCT01481766.

Biostatistics: essential concepts for the clinician

Introduction:

The efficiency of clinical procedures is based on practical and theoretical knowledge. Countless daily information is available to the orthodontist, but it is up to this professional to know how to select what really has an impact on clinical practice. Evidence-based orthodontics ends up requiring the clinician to know the basics of biostatistics to understand the results of scientific publications. Such concepts are also important for researchers, for correct data planning and analysis.

Objective:

This article aims to present, in a clear way, some essential concepts of biostatistics that assist the clinical orthodontist in understanding scientific research, for an evidence-based clinical practice. In addition, an updated version of the tutorial to assist in choosing the appropriate statistical test will be presented. This PowerPoint^® tool can be used to assist the user in finding answers to common questions about biostatistics, such as the most appropriate statistical test for comparing groups, choosing graphs, performing correlations and regressions, analyzing casual, random or systematic errors.

Conclusion:

Researchers and clinicians must acquire or recall essential concepts to understand and apply an appropriate statistical analysis. It is important that journal readers and reviewers can identify when statistical analyzes are being inappropriately used.

Essential statistical principles of clinical trials of pain treatments

This article presents an overview of fundamental statistical principles of clinical trials of pain treatments. Statistical considerations relevant to phase 2 proof of concept and phase 3 confirmatory randomized trials investigating efficacy and safety are discussed, including (1) research design; (2) endpoints and analyses; (3) sample size determination and statistical power; (4) missing data and trial estimands; (5) data monitoring and interim analyses; and (6) interpretation of results. Although clinical trials of pharmacologic treatments are emphasized, the key issues raised by these trials are also directly applicable to clinical trials of other types of treatments, including biologics, devices, nonpharmacologic therapies (eg, physical therapy and cognitive-behavior therapy), and complementary and integrative health interventions.

Interpretation of chronic pain clinical trial outcomes: IMMPACT recommended considerations

Interpreting randomized clinical trials (RCTs) is crucial to making decisions regarding the use of analgesic treatments in clinical practice. In this article, we report on an Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) consensus meeting organized by the Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks, the purpose of which was to recommend approaches that facilitate interpretation of analgesic RCTs. We review issues to consider when drawing conclusions from RCTs, as well as common methods for reporting RCT results and the limitations of each method. These issues include the type of trial, study design, statistical analysis methods, magnitude of the estimated beneficial and harmful effects and associated precision, availability of alternative treatments and their benefit-risk profile, clinical importance of the change from baseline both within and between groups, presentation of the outcome data, and the limitations of the approaches used.

Self-management program versus usual care for community-dwelling older adults with multimorbidity: A pragmatic randomized controlled trial in Ontario, Canada

Background:

Multimorbidity, the co-existence of 2+ (or 3+) chronic diseases in an individual, is an increasingly common global phenomenon leading to reduced quality of life and functional status, and higher healthcare service use and mortality. There is an urgent need to develop and test new models of care that incorporate the components of multimorbidity interventions recommended by international organizations, including care coordination, interdisciplinary teams, and care plans developed with patients that are tailored to their needs and preferences.

Purpose:

To determine the effectiveness of a 6-month, community-based, multimorbidity intervention compared to usual home care services for community-dwelling older adults (age 65+ years) with multimorbidity (3+ chronic conditions) that were newly referred to and receiving home care services.

Methods:

A pragmatic, parallel, two-arm randomized controlled trial evaluated the intervention, which included in-home visits by an interdisciplinary team, personal support worker visits, and monthly case conferences. The study took place in two sites in central Ontario, Canada. Eligible and consenting participants were randomly allocated to the intervention and control group using a 1:1 ratio. The participants, statistician/analyst, and research assistants collecting assessment data were blinded. The primary outcome was the Physical Component Summary (PCS) score of the 12-Item Short-Form health survey (SF-12). Secondary outcomes included the SF-12 Mental Component Summary (MCS) score, Center for Epidemiological Studies of Depression (CESD-10), Generalized Anxiety Disorder (GAD-7), Self-Efficacy for Managing Chronic Disease, and service use and costs. Analysis of covariance (ANCOVA) tested group differences using multiple imputation to address missing data, and non-parametric methods explored service use and cost differences.

Results:

59 older adults were randomized into the intervention (n = 30) and control (n = 29) groups. At baseline, groups were similar for the primary outcome and number of chronic conditions (mean of 8.6), but the intervention group had lower mental health status. The intervention was cost neutral and no significant group differences were observed for the primary outcome of PCS from SF-12 (mean difference: −4.94; 95% CI: −12.53 to 2.66; p = 0.20) or secondary outcomes.

Conclusion:

We evaluated a 6-month, self-management intervention for older adults with multimorbidity. While the intervention was cost neutral in comparison to usual care, it was not found to improve the PCS from SF-12 or secondary health outcomes. Recruitment and retention challenges were significant obstacles limiting our ability to assess intervention effectiveness. Yet, the intervention was grounded in internationally-endorsed recommendations and implemented in a practice setting (home care) viewed as a key upstream resource fostering independence in older adults. These features collectively support the identification of ways to recruit/retain older adults and test alternative implementation strategies for interventions that are based on sound principles of multimorbidity management.

Relief of Neuropathic Pain Through Epidermal Growth Factor Receptor Inhibition: A Randomized Proof-of-Concept Trial

OBJECTIVE

Case reports and a case series have described relief of neuropathic pain (NP) after treatment with epidermal growth factor receptor inhibitors (EGFR-Is). These observations are supported by preclinical findings. The aim of this trial was to explore a potential clinical signal supporting the therapeutic efficacy of EGFR-Is in NP.

METHODS

In a proof-of-concept trial using a randomized, double-blind, placebo-controlled design, 14 patients with severe, chronic, therapy-resistant NP due to compressed peripheral nerves or complex regional pain syndrome were randomized to receive a single infusion of the EGFR-I cetuximab and placebo in crossover design, followed by a single open-label cetuximab infusion.

RESULTS

The mean reduction in daily average pain scores three to seven days after single-blinded cetuximab infusion was 1.73 points (90% confidence interval [CI] = 0.80 to 2.66), conferring a 1.22-point greater reduction than placebo (90% CI = -0.10 to 2.54). Exploratory analyses suggested that pain reduction might be greater in the 14 days after treatment with blinded cetuximab than after placebo. The proportion of patients who reported ≥50% reduction in average pain three to seven days after cetuximab was 36% (14% after placebo), and comparison of overall pain reduction suggests a trend in favor of cetuximab. Skin rash (grade 1-2) was the most frequent side effect (12/14, 86%).

CONCLUSIONS

This small proof-of-concept evaluation of an EGFR-I against NP did not provide statistical evidence of efficacy. However, substantial reductions in pain were reported, and confidence intervals do not rule out a clinically meaningful treatment effect. Evaluation of EGFR-I against NP therefore warrants further investigation.

Effectiveness of a community program for older adults with type 2 diabetes and multimorbidity: a pragmatic randomized controlled trial

Background

Type II diabetes mellitus (T2DM) affects upwards of 25% of Canadian older adults and is associated with high comorbidity and burden. Studies show that lifestyle factors and self-management are associated with improved health outcomes, but many studies lack rigour or exclude older adults, particularly those with multimorbidity. More evidence is needed on the effectiveness of community-based self-management programs in older adults with T2DM and multimorbidity. The study purpose is to evaluate the effect of a community-based intervention versus usual care on physical functioning, mental health, depressive symptoms, anxiety, self-efficacy, self-management, and healthcare costs in older adults with T2DM and 2 or more comorbidities.

Methods

Community-living older adults with T2DM and two or more chronic conditions were recruited from three Primary Care Networks (PCNs) in Alberta, Canada. Participants were randomly allocated to the intervention or control group in this pragmatic randomized controlled trial comparing the intervention to usual care. The intervention involved up to three in-home visits, a monthly group wellness program, monthly case conferencing, and care coordination. The primary outcome was physical functioning. Secondary outcomes included mental functioning, anxiety, depressive symptoms, self-efficacy, self-management, and the cost of healthcare service use. Intention-to-treat analysis was performed using ANCOVA modeling.

Results

Of 132 enrolled participants (70-Intervention, 62-Control), 42% were 75 years or older, 55% were female, and over 75% had at least six chronic conditions (in addition to T2DM). No significant group differences were seen for the baseline to six-month change in physical functioning (mean difference: -0.74; 95% CI: − 3.22, 1.74; p-value: 0.56), mental functioning (mean difference: 1.24; 95% CI: − 1.12, 3.60; p-value: 0.30), or other secondary outcomes..

Conclusion

No significant group differences were seen for the primary outcome, physical functioning (PCS). Program implementation, baseline differences between study arms and chronic disease management services that are part of usual care may have contributed to the modest study results. Fruitful areas for future research include capturing clinical outcome measures and exploring the impact of varying the type and intensity of key intervention components such as exercise and diet.

Trial registration

NCT02158741 Date of registration: June 9, 2014.

An integrated hospital-to-home transitional care intervention for older adults with stroke and multimorbidity: A feasibility study

BACKGROUND

Stroke is the leading cause of death and adult disability in Canada. Eighty percent of older adults (≥65 years) who have suffered a stroke will return to their homes, and 60% will require ongoing rehabilitation. The transition between hospital and home is often fragmented, leading to adverse health outcomes, hospital readmissions, and increased health-care costs. This study examined the feasibility of a 6-month integrated transitional care stroke intervention (TCSI), and explored its effects on health outcomes, patient and provider experience, and cost in 30 community-living older adults (≥55 years) with stroke and multimorbidity (≥2 chronic conditions) using outpatient stroke rehabilitation services.

METHODS

The TCSI is a 6-month intervention delivered by an interprofessional (IP) team (occupational therapist, physiotherapist, speech language pathologist, registered nurse, social worker). It involved care coordination, home visiting, and IP case conferences, supported by a web-based application. A qualitative descriptive approach was used to explore the feasibility of implementing the intervention. A prospective one-group pretest/posttest was used to evaluate the effects of the intervention on health outcomes and use and costs of health services, from baseline to 6 months.

RESULTS

Participants had an average of eight comorbid conditions. The intervention was feasible and acceptable to both older adults and providers. From baseline to 6 months, there was no statistically significant difference in health outcomes. However, there was a significant reduction in the total per person use and costs of health services.

CONCLUSIONS

This study established the feasibility of conducting a larger randomized controlled trial of this intervention.

Reporting and interpretation of results from clinical trials that did not claim a treatment difference: survey of four general medical journals

OBJECTIVES

To describe and summarise how the results of randomised controlled trials (RCTs) that did not find a significant treatment effect are reported, and to estimate how commonly trial reports make unwarranted claims.

DESIGN

We performed a retrospective survey of published RCTs, published in four high impact factor general medical journals between June 2016 and June 2017.

SETTING

Trials conducted in all settings were included.

PARTICIPANTS

94 reports of RCTs that did not find a difference in their main comparison or comparisons were included.

INTERVENTIONS

All interventions.

PRIMARY AND SECONDARY OUTCOMES

We recorded the way the results of each trial for its primary outcome or outcomes were described in Results and Conclusions sections of the Abstract, using a 10-category classification. Other outcomes were whether confidence intervals (CIs) and p values were presented for the main treatment comparisons, and whether the results and conclusions referred to measures of uncertainty. We estimated the proportion of papers that made claims that were not justified by the results, or were open to multiple interpretations.

RESULTS

94 trial reports (120 treatment comparisons) were included. In Results sections, for 58/120 comparisons (48.3%) the results of the study were re-stated, without interpretation, and 38/120 (31.7%) stated that there was no statistically significant difference. In Conclusions, 65/120 treatment comparisons (54.2%) stated that there was no treatment benefit, 14/120 (11.7%) that there was no significant benefit and 16/120 (13.3%) that there was no significant difference. CIs and p values were both presented by 84% of studies (79/94), but only 3/94 studies referred to uncertainty when drawing conclusions.

CONCLUSIONS

The majority of trials (54.2%) inappropriately interpreted a result that was not statistically significant as indicating no treatment benefit. Very few studies interpreted the result as indicating a lack of evidence against the null hypothesis of zero difference between the trial arms.