Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials

High-impact trials with genetic and -omics information focus on cancer mutations, are industry-funded, and less transparent

OBJECTIVES

To assess how genetics and -omics information is used in the most cited recent clinical trials and to evaluate industry involvement and transparency patterns.

STUDY DESIGN AND SETTING

This is a meta-research evaluation using a previously constructed database of the 600 most cited clinical trials published from 2019 to 2022. Trials that utilized genetic or -omics characterization of participants in the trial design, analysis, and results were considered eligible.

RESULTS

132 (22%) trials used genetic or -omics information, predominantly for detection of cancer mutations (n = 101). Utilization included eligibility criteria (n = 59), subgroup analysis (n = 82), and stratification factor in randomization (n = 14). Authors addressed the relevance in the conclusions in 82 studies (62%). 102 studies (77%) provided data availability statements and six had data already available. Most studies had industry funding (n = 111 [84.0%]). Oncology trials were more likely to be industry-funded (90.1% vs 64.5%, P = .001), to have industry-affiliated analysts (43.6% vs 22.6%, P = .036), and to favor industry-sponsored interventions (83.2% vs 58.1% P = .004). When compared to other trials, genetic and -omics trials were more likely to be funded by industry (84% vs 63.9%, P < .001) and tended to be less likely to have full protocols (P = .018) and statistical plans (P = .04) available.

CONCLUSION

Our study highlights the current underutilization of genetic and -omics technologies beyond testing for cancer mutations. Industry involvement in these trials appears to be more substantial and transparency is more limited, raising concerns about potential bias.

Mixed-methods study to develop extensions to the SPIRIT and CONSORT statements for factorial randomised trials: the Reporting Factorial Trials (RAFT) study

BACKGROUND

Extensions to Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) and Consolidated Standards of Reporting Trials (CONSORT) reporting recommendations specifically for factorial trials have been developed by the Reporting Factorial Trials (RAFT) study group. This article describes the processes and methods used to develop the extensions.

OBJECTIVE

To develop SPIRIT and CONSORT extensions for factorial trials.

DESIGN AND PARTICIPANTS

A four-phase, consensus-based approach was used: phase 1: scoping review, phase 2: Delphi survey (n=104 respondents in round 1), phase 3: consensus meeting (n=15 members) and phase 4: checklist finalisation.

RESULTS

In phase 1, the scoping review identified 31 reporting recommendations, which formed a long list of 50 concepts (19 applied to the SPIRIT extension and 31 applied to the CONSORT extension) to include in the guideline development. In phase 2, a three-round Delphi survey resulted in two new concepts being added and ended with 49 concepts (19 applied to SPIRIT and 30 applied to CONSORT) reaching consensus to remain, with only three concepts meeting the exclusion criteria. In phase 3, the concepts were further refined and translated into specific extension item wording, through an extensive review process conducted by the core RAFT team and leading trial experts, who attended a 2-day hybrid meeting. The resulting 9 SPIRIT items and 17 CONSORT items were further evaluated and developed through an iterative process in phase 4, to promote user acceptance and uptake.

CONCLUSION

Uptake of the CONSORT and SPIRIT extensions will improve the conduct of factorial trials, as well as understanding and interpretation of such trials. By reporting on how these extensions were developed, we promote transparency of this process and share learning experiences to develop best practice when developing reporting guidelines.

Review of the quality of reporting of statistical analysis plans for cluster randomized trials

BACKGROUND AND OBJECTIVES

The guideline for the content of Statistical Analysis Plans (SAPs) outlines recommendations for items to be included in SAPs. As yet there is no specific tailoring of this guideline for Cluster Randomized Trials (CRTs). There has also been no assessment of reporting quality of SAPs against this guideline. Our intention is to identify how well a sample of SAPs for CRTs are adhering to the reporting of key items in the current guidelines, as well as additional analysis aspects considered to be important in CRTs.

METHODS

We include (i) fully published standalone SAPs identified via Ovid-MEDLINE and (ii) SAPs published as supplementary material or appendices to the final published report identified by searching an existing database of nearly 800 CRTs.

RESULTS

The search identified 85 unique SAPs: 26 were published in standalone format and 59 were supplementary material to the full trial report. There was mixed clarity in reporting of items related to the current guideline (eg, most (61/85, 72%) reported what covariates will be included in any adjustment; but fewer (26/85, 31%) reported what method will be used to estimate the absolute measure of effect). Considering additional aspects important for CRTs, the majority (79/85, 93%) included a plan to allow for clustering in the analysis; but fewer (10/40, 25%) reported how a small number of clusters would be accommodated (this was only considered relevant for the subset of CRTs with fewer than 40 clusters). Few (5/85, 6%) reported how the intracluster correlation would be estimated. Few clearly reported statistical targets of inference: in only two SAPs (2/85, 2%) it was clear whether the objectives were related to the individual or cluster-level average; in trials where relevant, only three (3/70, 4%) clearly reported whether the objectives were related to the marginal or cluster-specific effect.

CONCLUSION

This review has identified specific areas of poor quality of reporting that might need additional consideration when developing the guidance for the reporting of SAPs for CRTs.

Discrepancies in safety reporting for chronic back pain clinical trials: an observational study from ClinicalTrials.gov and publications

INTRODUCTION

Chronic back pain (CBP) is a leading cause of disability worldwide and is commonly managed with pharmacological, non-pharmacological, and procedural interventions. However, adverse event (AE) reporting for these therapies often lacks transparency, raising concerns about the accuracy of safety data. This study aimed to quantify inconsistencies in AE reporting between ClinicalTrials.gov and corresponding randomized controlled trial (RCT) publications, emphasizing the importance of comprehensive safety reporting to improve clinical decision-making and patient care.

METHODS

We retrospectively analyzed Phase 2-4 CBP RCTs registered on ClinicalTrials.gov from 2009 to 2023. Extracted data included AE reporting, trial sponsorship, and discrepancies in serious adverse events (SAEs), other adverse events (OAEs), mortality, and treatment-related withdrawals between registry entries and publications. Statistical analyses assessed reporting inconsistencies, following STROBE guidelines.

RESULTS

A total of 114 registered trials were identified, with 40 (35.1%) corresponding publications. Among these, 67.5% were industry-sponsored. Only 4 (10%) publications fully reported adverse events (AEs) without discrepancies, while 36 (90%) contained at least one inconsistency compared to ClinicalTrials.gov. Discontinuation due to AEs was explicitly reported in 24 (60%) of ClinicalTrials.gov entries and in 30 (75%) of publications, with discrepancies in 16 trials (40%). Serious adverse events (SAEs) were reported differently in 15 (37.5%) publications; 80% reported fewer SAEs than ClinicalTrials.gov. Other adverse events (OAEs) showed discrepancies in 37 (92.5%) publications, with 43.2% reporting fewer and 54.1% reporting more OAEs.

DISCUSSION

This study highlights pervasive discrepancies in AE reporting for CBP trials, undermining the reliability of published safety data. Inconsistent reporting poses risks to clinical decision-making and patient safety. Adopting standardized reporting guidelines, such as CONSORT Harms, and ensuring transparent updates in publications could enhance the accuracy and trustworthiness of safety data. Journals and regulatory bodies should enforce compliance and future efforts should develop mechanisms to monitor and correct reporting inconsistencies, enhancing the trustworthiness of safety data in clinical research.

How can we make sound replication decisions?

Replication and the reported crises impacting many fields of research have become a focal point for the sciences. This has led to reforms in publishing, methodological design and reporting, and increased numbers of experimental replications coordinated across many laboratories. While replication is rightly considered an indispensable tool of science, financial resources and researchers' time are quite limited. In this perspective, we examine different values and attitudes that scientists can consider when deciding whether to replicate a finding and how. We offer a conceptual framework for assessing the usefulness of various replication tools, such as preregistration.

Evaluating multiplicity reporting in analgesic clinical trials: An analytical review

BACKGROUND AND OBJECTIVES

Analgesia trials often demands multiple comparisons to assess various treatment arms, outcomes, or repeated assessments. These multiple comparisons risk inflating the false positive rate. Multiplicity correction in recent analgesic randomized controlled trials (RCTs) remains unclear despite statistical method advancements and regulatory guidelines. Our study aimed to identify reporting inadequacies in multiple analysis adjustments and explanations to understand these deficiencies.

DATABASES AND DATA TREATMENT

This review analysed RCTs from the European Journal of Pain, the Journal of Pain, and PAIN, published between January 2018 and December 2022. We included randomized, double-blind trials focusing on pain outcomes. Data extraction, managed by three researchers using predefined criteria, included trial characteristics, multiplicity presence, and correction methods. Descriptive statistical analyses included Fisher's exact, and Holm method for multiple comparisons.

RESULTS

Out of 112 articles, 48 pre-specified a primary analysis plan. Multiple analyses were observed in 65 articles, with 60% adjusting for all comparisons, primarily using the Bonferroni method. Compared with previous studies, no significant changes in multiplicity correction practices were noted when stratified by trial type, size, and sponsor.

CONCLUSIONS

The study reveals a persistent reliance on multiple comparisons in analgesic clinical trials without a corresponding increase in multiplicity corrections emphasizing a need for enhanced reporting and implementation of statistical adjustments. We acknowledge limitations in categorizing studies, the use of a surrogate for the trial stage, and sourcing data from journal webpages rather than a database.

SIGNIFICANCE STATEMENT

This study flags inadequate reporting on multiplicity correction in analgesic trials, stressing the risk of false positives and the urgent need for enhanced reporting to boost reproducibility.

Occlusal outcome of orthodontic treatment: a systematic review with meta-analyses of randomized trials

BACKGROUND

Several appliances or treatment protocols are marketed to either patients or orthodontists as being associated with improved orthodontic outcomes. However, clinical decision-making should be based on robust scientific evidence and not marketing claims or anecdotal evidence.

OBJECTIVE

To identify appliances/protocols being associated with improved outcomes of fixed appliance treatment.

SEARCH METHODS

Unrestricted literature searches in seven databases/registers for human studies until March 2024.

SELECTION CRITERIA

Randomized or quasi-randomized clinical trials on human patients of any age, sex, or ethnicity receiving comprehensive orthodontic treatment with fixed appliances and assessing occlusal outcome with either the Peer Assessment Rating (PAR) or the American Board of Orthodontics-Objective Grading System (ABO-OGS) index.

DATA COLLECTION AND ANALYSIS

Duplicate/independent study selection, data extraction, and risk of bias assessment with the Cochrane RoB 2 tool. Random-effects meta-analyses of averages or mean differences with their 95% Confidence Intervals (CI), followed by meta-regression/subgroup/sensitivity analyses and assessment of the quality of clinical recommendations with the Grades of Recommendations, Assessment, Development, and Evaluation (GRADE) approach.

RESULTS

Data from 20 small- to moderately-sized trials covering 1470 patients indicated that orthodontic treatment with fixed appliances is effective and results on average in a final PAR score of 6.0 points (95% CI 3.9-8.2 points), an absolute PAR reduction of 23.0 points (95% CI 15.6-30.4 points), a % PAR reduction of 82.6% (95% CI 70.8%-94.4%), and an absolute ABO-OGS score of 18.9 points (95% CI 11.7-26.2 points). However, very high between-study heterogeneity (I2 > 75%) was seen for both PAR and ABO-OGS. Extraction treatment was associated with significantly better occlusal outcome than non-extraction treatment with ABO-OGS (12.9 versus 16.6 points; P = .02). There was no statistically significant difference in occlusal outcome with (i) 0.018″-slot or 0.022″-slot brackets; (ii) customized or prefabricated brackets; (iii) anchorage reinforcement with temporary anchorage devices; (iv) use of vibrational adjuncts; and (v) aligners or fixed appliances (P > .05 in all instances), while small benefits were seen with indirectly bonded brackets.

CONCLUSIONS

Considerable between-study heterogeneity exists in the reported occlusal outcome of fixed appliance treatment, and different appliances or adjuncts have little effect on this. Standardization and/or automatization of the scoring procedures for PAR and ABO-OGS might help to improve consistency and reliability of outcome measurement in orthodontic trials.

REGISTRATION

PROSPERO (CRD42024525088).

Industry Involvement and Transparency in the Most Cited Clinical Trials, 2019-2022

Importance

Industry involvement is prominent in influential clinical trials, and commitments to transparency of trials are highly variable.

Objective

To evaluate the modes of industry involvement and the transparency features of the most cited recent clinical trials across medicine.

Design, Setting, and Participants

This cross-sectional study was a meta-research assessment including randomized and nonrandomized clinical trials published in 2019 or later. The 600 trials of any type of disease or setting that attracted highest number of citations in Scopus as of December 2022 were selected for analysis. Data were analyzed from March to September 2023.

Main Outcomes and Measures

Outcomes of interest were industry involvement (sponsor, author, and analyst) and transparency (protocols, statistical analysis plans, and data and code availability).

Results

Among 600 trials with a median (IQR) sample size of 415 (124-1046) participants assessed, 409 (68.2%) had industry funding and 303 (50.5%) were exclusively industry-funded. A total of 354 trials (59.0%) had industry authors, with 280 trials (46.6%) involving industry analysts and 125 trials (20.8%) analyzed exclusively by industry analysts. Among industry-funded trials, 364 (89.0%) reached conclusions favoring the sponsor. Most trials (478 trials [79.7%]) provided a data availability statement, and most indicated intention to share the data, but only 16 trials (2.7%) had data already readily available to others. More than three-quarters of trials had full protocols (482 trials [82.0%]) or statistical analysis plans (446 trials [74.3%]) available, but only 27 trials (4.5%) explicitly mentioned sharing analysis code (8 readily available; 19 on request). Randomized trials were more likely than nonrandomized studies to involve only industry analysts (107 trials [22.9%] vs 18 trials [13.6%]; P = .02) and to have full protocols (405 studies [86.5%] vs 87 studies [65.9%]; P < .001) and statistical analysis plans (373 studies [79.7%] vs 73 studies [55.3%]; P < .001) available. Almost all nonrandomized industry-funded studies (90 of 92 studies [97.8%]) favored the sponsor. Among industry-funded trials, exclusive industry funding (odds ratio, 2.9; 95% CI, 1.5-5.4) and industry-affiliated authors (odds ratio, 2.9; 95% CI, 1.5-5.6) were associated with favorable conclusions for the sponsor.

Conclusions and Relevance

This cross-sectional study illustrates how industry involvement in the most influential clinical trials was prominent not only for funding, but also authorship and provision of analysts and was associated with conclusions favoring the sponsor. While most influential trials reported that they planned to share data and make both protocols and statistical analysis plans available, raw data and code were rarely readily available.

A template for the authoring of statistical analysis plans

A number of principal investigators may have limited access to biostatisticians, a lack of biostatistical training, or no requirement to complete a timely statistical analysis plan (SAP). SAPs completed early will identify design or implementation weak points, improve protocols, remove the temptation for p-hacking, and enable proper peer review by stakeholders considering funding the trial. An SAP completed at the same time as the study protocol might be the only comprehensive method for at once optimizing sample size, identifying bias, and applying rigor to study design. This ordered corpus of SAP sections with detailed definitions and a variety of examples represents an omnibus of best practice methods offered by biostatistical practitioners inside and outside of industry. The article presents a protocol template for clinical research design, enabling statisticians, from beginners to advanced.

Eliminating Ambiguous Treatment Effects Using Estimands

Most reported treatment effects in medical research studies are ambiguously defined, which can lead to misinterpretation of study results. This is because most authors do not attempt to describe what the treatment effect represents, and instead require readers to deduce this based on the reported statistical methods. However, this approach is challenging, because many methods provide counterintuitive results. For example, some methods include data from all patients, yet the resulting treatment effect applies only to a subset of patients, whereas other methods will exclude certain patients while results will apply to everyone. Additionally, some analyses provide estimates pertaining to hypothetical settings in which patients never die or discontinue treatment. Herein we introduce estimands as a solution to the aforementioned problem. An estimand is a clear description of what the treatment effect represents, thus saving readers the necessity of trying to infer this from study methods and potentially getting it wrong. We provide examples of how estimands can remove ambiguity from reported treatment effects and describe their current use in practice. The crux of our argument is that readers should not have to infer what investigators are estimating; they should be told explicitly.

Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands

OBJECTIVES

To evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials.

DESIGN

Systematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals.

DATA SOURCE

PubMed search in February 2021.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Phase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded.

MAIN OUTCOME MEASURES

Number of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated.

RESULTS

255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)-that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die).

CONCLUSIONS

The precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42021238053.

Access to unpublished protocols and statistical analysis plans of randomised trials

BACKGROUND

Access to protocols and statistical analysis plans (SAPs) increases the transparency of randomised trial by allowing readers to identify and interpret unplanned changes to study methods, however they are often not made publicly available. We sought to determine how often study investigators would share unavailable documents upon request.

METHODS

We used trials from two previously identified cohorts (cohort 1: 101 trials published in high impact factor journals between January and April of 2018; cohort 2: 100 trials published in June 2018 in journals indexed in PubMed) to determine whether study investigators would share unavailable protocols/SAPs upon request. We emailed corresponding authors of trials with no publicly available protocol or SAP up to four times.

RESULTS

Overall, 96 of 201 trials (48%) across the two cohorts had no publicly available protocol or SAP (11/101 high-impact cohort, 85/100 PubMed cohort). In total, 8/96 authors (8%) shared some trial documentation (protocol only [n = 5]; protocol and SAP [n = 1]; excerpt from protocol [n = 1]; research ethics application form [n = 1]). We received protocols for 6/96 trials (6%), and a SAP for 1/96 trial (1%). Seventy-three authors (76%) did not respond, 7 authors responded (7%) but declined to share a protocol or SAP, and eight email addresses were invalid (8%). A total of 329 emails were sent (an average of 41 emails for every trial which sent documentation). After emailing authors, the total number of trials with an available protocol increased by only 3%, from 52% in to 55%.

CONCLUSIONS

Most study investigators did not share their unpublished protocols or SAPs upon direct request. Alternative strategies are needed to increase transparency of randomised trials and ensure access to protocols and SAPs.

Treatment estimands in clinical trials of patients hospitalised for COVID-19: ensuring trials ask the right questions

When designing a clinical trial, explicitly defining the treatment estimands of interest (that which is to be estimated) can help to clarify trial objectives and ensure the questions being addressed by the trial are clinically meaningful. There are several challenges when defining estimands. Here, we discuss a number of these in the context of trials of treatments for patients hospitalised with COVID-19 and make suggestions for how estimands should be defined for key outcomes. We suggest that treatment effects should usually be measured as differences in proportions (or risk or odds ratios) for outcomes such as death and requirement for ventilation, and differences in means for outcomes such as the number of days ventilated. We further recommend that truncation due to death should be handled differently depending on whether a patient- or resource-focused perspective is taken; for the former, a composite approach should be used, while for the latter, a while-alive approach is preferred. Finally, we suggest that discontinuation of randomised treatment should be handled from a treatment policy perspective, where non-adherence is ignored in the analysis (i.e. intention to treat).

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework

Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and ICH-E9 (International Conference for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use) require the statistical methods for a trial's primary outcome be pre-specified in the trial protocol. However, pre-specification is only effective if done in a way that does not allow p-hacking. For example, investigators may pre-specify a certain statistical method such as multiple imputation, but give little detail on how it will be implemented. Because there are many different ways to perform multiple imputation, this approach to pre-specification is ineffective, as it still allows investigators to analyse the data in different ways before deciding on a final approach. In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking. This framework was designed based on the principles in the SPIRIT and ICH-E9 guidelines and is intended to be used in conjunction with these guidelines to help investigators design the statistical analysis strategy for the trial's primary outcome in the trial protocol.

Public availability and adherence to prespecified statistical analysis approaches was low in published randomized trials

BACKGROUND AND OBJECTIVE

Prespecification of statistical methods in clinical trial protocols and statistical analysis plans can help to deter bias from p-hacking but is only effective if the prespecified approach is made available.

STUDY DESIGN AND SETTING

For 100 randomized trials published in 2018 and indexed in PubMed, we evaluated how often a prespecified statistical analysis approach for the trial's primary outcome was publicly available. For each trial with an available prespecified analysis, we compared this with the trial publication to identify whether there were unexplained discrepancies.

RESULTS

Only 12 of 100 trials (12%) had a publicly available prespecified analysis approach for their primary outcome; this document was dated before recruitment began for only two trials. Of the 12 trials with an available prespecified analysis approach, 11 (92%) had one or more unexplained discrepancies. Only 4 of 100 trials (4%) stated that the statistician was blinded until the SAP was signed off, and only 10 of 100 (10%) stated the statistician was blinded until the database was locked.

CONCLUSION

For most published trials, there is insufficient information available to determine whether the results may be subject to p-hacking. Where information was available, there were often unexplained discrepancies between the prespecified and final analysis methods.