A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials

Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses

OBJECTIVES

Prospectively registering study plans in a permanent time-stamped and publicly accessible document is becoming more common across disciplines and aims to reduce risk of bias and make risk of bias transparent. Selective reporting persists, however, when researchers deviate from their registered plans without disclosure. This systematic review aimed to estimate the prevalence of undisclosed discrepancies between prospectively registered study plans and their associated publication. We further aimed to identify the research disciplines where these discrepancies have been observed, whether interventions to reduce discrepancies have been conducted, and gaps in the literature.

DESIGN

Systematic review and meta-analyses.

DATA SOURCES

Scopus and Web of Knowledge, published up to 15 December 2019.

ELIGIBILITY CRITERIA

Articles that included quantitative data about discrepancies between registrations or study protocols and their associated publications.

DATA EXTRACTION AND SYNTHESIS

Each included article was independently coded by two reviewers using a coding form designed for this review (osf.io/728ys). We used random-effects meta-analyses to synthesise the results.

RESULTS

We reviewed k=89 articles, which included k=70 that reported on primary outcome discrepancies from n=6314 studies and, k=22 that reported on secondary outcome discrepancies from n=1436 studies. Meta-analyses indicated that between 29% and 37% (95% CI) of studies contained at least one primary outcome discrepancy and between 50% and 75% (95% CI) contained at least one secondary outcome discrepancy. Almost all articles assessed clinical literature, and there was considerable heterogeneity. We identified only one article that attempted to correct discrepancies.

CONCLUSIONS

Many articles did not include information on whether discrepancies were disclosed, which version of a registration they compared publications to and whether the registration was prospective. Thus, our estimates represent discrepancies broadly, rather than our target of undisclosed discrepancies between prospectively registered study plans and their associated publications. Discrepancies are common and reduce the trustworthiness of medical research. Interventions to reduce discrepancies could prove valuable.

REGISTRATION

osf.io/ktmdg. Protocol amendments are listed in online supplemental material A.

Evaluation of randomized controlled trials: a primer and tutorial for mental health researchers

BACKGROUND

Considered one of the highest levels of evidence, results of randomized controlled trials (RCTs) remain an essential building block in mental health research. They are frequently used to confirm that an intervention "works" and to guide treatment decisions. Given their importance in the field, it is concerning that the quality of many RCT evaluations in mental health research remains poor. Common errors range from inadequate missing data handling and inappropriate analyses (e.g., baseline randomization tests or analyses of within-group changes) to unduly interpretations of trial results and insufficient reporting. These deficiencies pose a threat to the robustness of mental health research and its impact on patient care. Many of these issues may be avoided in the future if mental health researchers are provided with a better understanding of what constitutes a high-quality RCT evaluation.

METHODS

In this primer article, we give an introduction to core concepts and caveats of clinical trial evaluations in mental health research. We also show how to implement current best practices using open-source statistical software.

RESULTS

Drawing on Rubin's potential outcome framework, we describe that RCTs put us in a privileged position to study causality by ensuring that the potential outcomes of the randomized groups become exchangeable. We discuss how missing data can threaten the validity of our results if dropouts systematically differ from non-dropouts, introduce trial estimands as a way to co-align analyses with the goals of the evaluation, and explain how to set up an appropriate analysis model to test the treatment effect at one or several assessment points. A novice-friendly tutorial is provided alongside this primer. It lays out concepts in greater detail and showcases how to implement techniques using the statistical software R, based on a real-world RCT dataset.

DISCUSSION

Many problems of RCTs already arise at the design stage, and we examine some avoidable and unavoidable "weak spots" of this design in mental health research. For instance, we discuss how lack of prospective registration can give way to issues like outcome switching and selective reporting, how allegiance biases can inflate effect estimates, review recommendations and challenges in blinding patients in mental health RCTs, and describe problems arising from underpowered trials. Lastly, we discuss why not all randomized trials necessarily have a limited external validity and examine how RCTs relate to ongoing efforts to personalize mental health care.

PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews

This is the protocol for a Campbell systematic review. The objectives are as follows: To identify methods used to assess the risk of outcome reporting bias (ORB) in studies included in recent Campbell systematic reviews of intervention effects. The review will answer the following questions: What proportion of recent Campbell reviews included assessment of ORB? How did recent reviews define levels of risk of ORB (what categories, labels, and definitions did they use)? To what extent and how did these reviews use study protocols as sources of data on ORB? To what extent and how did reviews document reasons for judgments about risk of ORB? To what extent and how did reviews assess the inter-rater reliability of ORB ratings? To what extent and how were issues of ORB considered in the review's abstract, plain language summary, and conclusions?

Selective outcome reporting in paediatric dentistry restorative treatment randomised clinical trials-A meta-research

BACKGROUND

Selective outcome reporting (SOR) is a bias that occurs when the primary outcome of a randomised clinical trial (RCT) is omitted or changed.

AIM

To evaluate the prevalence of SOR in RCTs on restorative treatment in primary teeth.

DESIGN

We conducted an electronic search on ClinicalTrials.gov and the World Health Organization platform (International Clinical Trials Registry Platform) on 1^st of April 2021, with no registry time or language restrictions. We included RCT protocols that evaluated restorative treatments in primary teeth and excluded trials that did not have a complete publication in a scientific journal. The chi-squared test was used to identify the association between SOR and variables as a discrepancy in the follow-up period, the timing of registration, the type of sponsorship and the type of study design (α = 5%).

RESULTS

Of the 294 identified protocols, 30 were included in the study. 83.3% of trials were registered retrospectively. SOR was observed in 53.3% (n = 16) of the published trials and was significantly associated with a discrepancy in the follow-up period (p = .017).

CONCLUSIONS

The high prevalence of SOR in RCTs on restorative treatment proves that this is a prominent threat. A proper preregistered protocol, declaration of any protocol deviation and allowance of stakeholders to compare the protocol with that of the submitted papers will achieve transparency.

The Campbell Collaboration's systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards

BACKGROUND

Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration's mandatory methodological standards.

ISSUES

In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review.

CONCLUSIONS

Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

Registry versus publication: discrepancy of primary outcomes and possible outcome reporting bias in child and adolescent mental health

Outcome reporting bias is one of the fundamental forms of publication bias. It implies publishing only outcomes that have positive results. The aim of this observational study was to explore primary outcome discrepancies between registry of clinical trials and their corresponding publications, since these can indicate outcome reporting bias in child mental health. Data were extracted from completed interventional clinical trials from ClinicalTrial.gov registry and its Archive site. Trials were registered under "Behaviours and Mental Disorders" category, and conducted on underage participants (0-17 years). Their primary outcomes were compared to those published in publication which had a corresponding NCT number stated in the text. Sixteen percent of trials did not have the minimum information on primary outcome stated in the registry-neither the measure used nor the measurement time points; 38.9% of trials had the minimum information stated to describe primary outcome, while only 3.3% of trials had all the necessary elements stated in the registry. Most of the publication in our sample had positive results (66.4%). Half of the trials registered before completion had non-matching primary outcomes in the registry and publication; 85.4% of trials with non-matching outcomes indicated possible outcome reporting bias for some of the primary outcome. Middle-sized trials and industry-funded trials were related with higher quality of primary outcome registration. Industry funding was related with positive findings in publication. Non-industry funding proved to be the only significant predictor of discrepancy between registered and published primary outcomes, and possible outcome reporting bias. Journal impact factor was not related with any of the outcome measures. The main limitation of the study is that it primarily offers an insight into discrepancy of registered and published outcomes. The methodology does not imply an access to results of unpublished outcomes - therefore, it was not possible to determine the presence of the bias with sufficient certainty in large number of trials. Further research should be done with improved methodology and additional data.

Contingency management for treatment attendance: A meta-analysis

BACKGROUND

Treatment providers have applied contingency management (CM) treatment, an intervention that often rewards individuals for drug abstinence (i.e., ABS CM), to treatment engagement as well. However, we know little about the magnitude of treatment effects when providers target attendance behaviors (i.e., ATT CM).

METHODS

This study conducted a systematic search to identify studies that included ATT CM, either in isolation or in combination with ABS CM. The study used meta-analysis to estimate the effect size of ATT CM and ABS CM + ATT CM on treatment attendance and drug abstinence. We identified a total of 10 studies including 12 CM treatments (6 ATT CM and 6 ABS CM + ATT CM) with 1841 participants.

RESULTS

Results indicated a moderate effect (d = 0.47, 95% confidence interval (CI) [0.25, 0.69]) of ATT CM on attendance relative to non-reward active comparison conditions. Frequency of rewards was significantly associated with larger effect sizes. Results also indicated a small effect (d = 0.22, 95% CI [0.12, 0.33]) of ATT CM on abstinence outcomes relative to nonreward comparisons, p < 0.001. The study found no significant differences in attendance or abstinence between ATT CM and ABS CM + ATT CM (p's > 0.05).

CONCLUSION

Overall, the results supported ATT CM for increasing treatment engagement, with smaller effects on abstinence. Effects on abstinence were smaller than those observed in prior meta-analyses focused on ABS CM. No significant differences existed in attendance or abstinence outcomes between ATT CM and ABS + ATT CM. However, future studies are needed to experimentally compare ABS CM + ATT CM to ABS CM, and ATT CM to determine additive effects. Clinics implementing CM should consider the differential effects between ATT CM and ABS CM when selecting target behavior(s).

Re-Analyzing Phase III Bremelanotide Trials for "Hypoactive Sexual Desire Disorder" in Women

Kingsberg et al. described results from two 24-week Phase III trials of bremelanotide for treating hypoactive sexual desire disorder (HSDD) in women. 72.72% of protocol-listed outcomes were not reported by Kingsberg et al., who provided results of 15 secondary measures which were not listed in the study protocols. None of their efficacy outcomes were reported in line with CONSORT data reporting standards and no secondary outcome had a stated rationale or cited evidence of validity. My meta-analysis of the trials' data, based on the FDA New Drug Application, found similar results to Kingsberg et al. However, Kingsberg et al. did not report that a) adverse event-induced study discontinuation was substantially higher on bremelanotide: OR = 11.98, 95% CI = 3.74-38.37, NNH: 6 or b) participants preferred placebo, measured by the combination of both 1) completing a clinical trial and 2) electing to participate in the follow-up open-label study (OR = 0.30, 95% CI = .24-.38, NNH: 4). Bremelanotide's modest benefits on incompletely reported post-hoc measures of questionable validity in combination with participants substantially preferring to take placebo suggest that the drug is generally not useful. Kingsberg et al.'s data reporting and measurement practices were incomplete and lacked transparency.

Trial registration and selective outcome reporting in 585 clinical trials investigating drugs for prevention of postoperative nausea and vomiting

Background

Selective outcome reporting in clinical trials introduces bias in the body of evidence distorting clinical decision making. Trial registration aims to prevent this bias and is suggested by the International Committee of Medical Journal Editors (ICMJE) since 2004.

Methods

The 585 randomized controlled trials (RCTs) published between 1965 and 2017 that were included in a recently published Cochrane review on antiemetic drugs for prevention of postoperative nausea and vomiting were selected. In a retrospective study, we assessed trial registration and selective outcome reporting by comparing study publications with their registered protocols according to the ‘Cochrane Risk of bias’ assessment tool 1.0.

Results

In the Cochrane review, the first study which referred to a registered trial protocol was published in 2004. Of all 585 trials included in the Cochrane review, 334 RCTs were published in 2004 or later, of which only 22% (75/334) were registered. Among the registered trials, 36% (27/75) were pro- and 64% (48/75) were retrospectively registered. 41% (11/27) of the prospectively registered trials were free of selective outcome reporting bias, 22% (6/27) were incompletely registered and assessed as unclear risk, and 37% (10/27) were assessed as high risk. Major outcome discrepancies between registered and published high risk trials were a change from the registered primary to a published secondary outcome (32%), a new primary outcome (26%), and different outcome assessment times (26%). Among trials with high risk of selective outcome reporting 80% favoured at least one statistically significant result. Registered trials were assessed more often as ‘overall low risk of bias’ compared to non-registered trials (64% vs 28%).

Conclusions

In 2017, 13 years after the ICMJE declared prospective protocol registration a necessity for reliable clinical studies, the frequency and quality of trial registration in the field of PONV is very poor. Selective outcome reporting reduces trustworthiness in findings of clinical trials. Investigators and clinicians should be aware that only following a properly registered protocol and transparently reporting of predefined outcomes, regardless of the direction and significance of the result, will ultimately strengthen the body of evidence in the field of PONV research in the future.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12871-021-01464-w.

Long-term efficacy of contingency management treatment based on objective indicators of abstinence from illicit substance use up to 1 year following treatment: A meta-analysis

OBJECTIVE

Contingency management (CM) is often criticized for limited long-term impact. This meta-analysis focused on objective indices of drug use (i.e., urine toxicology) to examine the effects of CM on illicit substance use up to 1 year following treatment.

METHOD

Analyses included randomized trials (k = 23) of CM for stimulant, opioid, or polysubstance use disorders that reported outcomes up to 1 year after the incentive delivery had ended. Using random effects models, odds ratios (OR) were calculated for the likelihood of abstinence. Metaregressions and subgroup analyses explored how parameters of CM treatment, namely escalation, frequency, immediacy, and magnitude of reinforcers, moderated outcomes.

RESULTS

The overall likelihood of abstinence at the long-term follow-up among participants who received CM versus a comparison treatment (nearly half of which were community-based comprehensive therapies or protocol-based specific therapies) was OR = 1.22, 95% confidence interval [1.01, 1.44], with low to moderate heterogeneity (I² = 36.68). Among 18 moderators, longer length of active treatment was found to significantly improve long-term abstinence.

CONCLUSIONS

CM showed long-term benefit in reducing objective indices of drug use, above and beyond other active, evidence-based treatments (e.g., cognitive-behavioral therapy, 12-step facilitation) and community-based intensive outpatient treatment. These data suggest that policymakers and insurers should support and cover costs for CM, which is the focus of hundreds of studies demonstrating its short-term efficacy and, now, additional data supporting its long-term efficacy. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Ethical Publication Standards in Articles Reporting on Novel Surgical Methods: Analysis of Three Pediatric Surgical Journals

INTRODUCTION

 According to the Declaration of Helsinki, medical research and new therapeutic interventions involving human subjects require prior informed consent and ethical approval. In 2010, 46% of pediatric surgical publications lacked documentation of ethical approval and 84% lacked documentation of informed parental consent with lowest rates of ethical adherence found in articles concerning novel methods. The aim of this study was to investigate whether adherence to ethical standards has improved in pediatric surgical publications.

MATERIALS AND METHODS

 All 3,093 consecutive articles published in Journal of Pediatric Surgery, European Journal of Pediatric Surgery, and Pediatric Surgery International over the last 5 years were systematically reviewed for publications describing novel surgical methods. Novel methods were defined as surgical methods not published before or not considered common practice. The publications were reviewed as to whether ethical approval and informed consent to participate was documented.

RESULTS

 In total, 105 articles describing novel surgical methods were identified (61 Journal of Pediatric Surgery, 16 European Journal of Pediatric Surgery, and 28 Pediatric Surgery International). Authors reported on new operative techniques (62%), modified techniques (31%), or use of new materials (7%). Ethical approval was documented in 52% of the articles with almost half reporting approval for retrospective data analysis only but not the application of the novel method. Informed consent was documented in 21% of publications. Complications were reported in 48% of the studies, including recurrences and reinterventions for the unsuccessful novel methods. Two authors reported mortalities due to underlying disease, one of which failed to report prior ethical approval or informed consent.

CONCLUSION

 Adherence to ethical publication principles in pediatric surgery has improved over the last years but is still lacking in many publications. When implementing new methods, prior ethical approval and informed consent and their documentation are mandatory, specifically in the light of potential hazard to patients.

Trial Registration and Outcome Reporting in Child and Pediatric Psychology: A Systematic Review

OBJECTIVE

To examine rate of registration for randomized controlled trials (RCTs) published in the Journal of Pediatric Psychology (JPP) and Journal of Clinical Child & Adolescent Psychology (JCCAP). Secondary aims were to investigate associations between trial characteristics and registration status and compare registered and published primary outcomes.

METHODS

RCTs published in JPP or JCCAP between January 1, 2007 and December 31, 2017 were included. Secondary analyses of previously published RCTs, meta-analytic, systematic, and narrative reviews, and articles reporting primary aims related to intervention acceptability, feasibility, and/or cost-effectiveness were excluded. Trial registration status, primary registered and published outcomes, dates of registration, participant enrollment and publication, sample size, and country where the trial was conducted were extracted from articles and trial registries.

RESULTS

Of 61 RCTs included, 48% were registered. Among registered trials, only 14% were registered before participant enrollment began. Most were registered late (i.e., retrospectively; 86%) in ClinicalTrials.gov (90%). Registration status did not differ based on journal, study sample size, or geographic region where the study was conducted. A greater proportion of trials published in 2013-2017 were registered (61%) relative to those published in 2007-2012 (32%), p = .03. Among registered trials, 57% had discrepancies between registered and published primary outcomes.

CONCLUSIONS

Findings reveal low rates of prospective registration and considerable risk for incomplete or selective outcome reporting among RCTs published in JPP and JCCAP. Coordinated efforts from all stakeholders involved in the conduct and reporting of clinical child and pediatric psychology research are needed to improve transparent reporting of clinical trials.

Evaluation of selective outcome reporting and trial registration practices among addiction clinical trials

BACKGROUND AND AIMS

Selective outcome reporting occurs when trialists pre-specify primary and secondary outcomes during trial planning but alter the definitions in the published report. Here, we investigate selective outcome reporting in published addiction randomized controlled trials (RCTs) and evaluate whether particular funding sources are associated with an increased likelihood of selective outcome reporting.

DESIGN

We conducted a cross-sectional study of published addiction clinical trials. A PubMed search was performed to identify RCTs in addiction journals from 2013 to 2017. Included studies used a randomized design to address one of the following topics: (1) drug, alcohol and tobacco addiction prevention, (2) stabilization following excessive use of a substance, (3) relapse prevention or (4) recovery maintenance.

SETTING

Single-center, medical research institution.

PARTICIPANTS

Our sample included 162 RCTs that were prospectively registered with a clearly defined primary outcome.

MEASUREMENT

We extracted the following items from addiction RCTs: journal, funding source, trial registry number (if included), sample size, dates of subject enrollment, whether primary and secondary outcomes were denoted, all published outcomes, P-value for all outcomes and whether authors mentioned any deviations from the trial protocol as it related to RCT outcomes.

FINDINGS

In total, 47 of 162 RCTs (29.0%) had at least one major discrepancy between the trial registry and published RCT. Overall, these 47 RCTs included 54 major discrepancies. The most common major discrepancy was demotion of a primary registered outcome (19/54, 35.2%). The majority of RCTs (132/162, 81.5%) were funded from public sources. Additionally, 166 RCTs were excluded from our sample because registration could not be confirmed.

CONCLUSIONS

There is evidence suggestive of selective outcome reporting in addiction randomized controlled trials (RCTs). The most common major discrepancies pertained to the primary outcome.

Prospective registration and reporting of trial number in randomised clinical trials: global cross sectional study of the adoption of ICMJE and Declaration of Helsinki recommendations

OBJECTIVES

To evaluate the compliance with prospective registration and inclusion of the trial registration number (TRN) in published randomised controlled trials (RCTs), and to analyse the rationale behind, and detect selective registration bias in, retrospective trial registration.

DESIGN

Cross sectional analysis.

DATA SOURCES

PubMed, the 17 World Health Organization's trial registries, University of Toronto library, International Committee of Medical Journal Editors (ICMJE) list of member journals, and the InCites Journal Citation Reports.

STUDY SELECTION CRITERIA

RCTs registered in any WHO trial registry and published in any PubMed indexed journal in 2018.

RESULTS

This study included 10 500 manuscripts published in 2105 journals. Overall, 71.2% (7473/10500) reported the TRN and 41.7% (3013/7218) complied with prospective trial registration. The univariable and multivariable analyses reported significant relations (P<0.05) between reporting the TRN and the impact factor and ICMJE membership of the publishing journal. A significant relation (P<0.05) was also observed between prospective trial registration and the registry, region, condition, funding, trial size, interval between paper registration and submission dates, impact factor, and ICMJE membership of the publishing journal. A manuscript published in an ICMJE member journal was 5.8 times more likely to include the TRN (odds ratio 5.8, 95% confidence interval 4.0 to 8.2), and a published trial was 1.8 times more likely to be registered prospectively (1.8, 1.5 to 2.2) when published in an ICMJE member journal compared with other journals. This study detected a new form of bias, selective registration bias, with a higher proportion (85.2% (616/723)) of trials registered retrospectively within a year of submission for publication. Higher rates of retrospective registrations were observed within the first three to eight weeks after enrolment of study participants. Within the 286 RCTs registered retrospectively and published in an ICMJE member journal, only 2.8% (8/286) of the authors included a statement justifying the delayed registration. Reasons included lack of awareness, error of omission, and the registration process taking longer than anticipated.

CONCLUSIONS

This study found a high compliance in reporting of the TRN for trial papers published in ICMJE member journals, but prospective trial registration was low.

Assessment of transparent and reproducible research practices in the psychiatry literature

BACKGROUND

Reproducibility is a cornerstone of scientific advancement; however, many published works may lack the core components needed for study reproducibility.

AIMS

In this study, we evaluate the state of transparency and reproducibility in the field of psychiatry using specific indicators as proxies for these practices.

METHODS

An increasing number of publications have investigated indicators of reproducibility, including research by Harwicke et al, from which we based the methodology for our observational, cross-sectional study. From a random 5-year sample of 300 publications in PubMed-indexed psychiatry journals, two researchers extracted data in a duplicate, blinded fashion using a piloted Google form. The publications were examined for indicators of reproducibility and transparency, which included availability of: materials, data, protocol, analysis script, open-access, conflict of interest, funding and online preregistration.

RESULTS

This study ultimately evaluated 296 randomly-selected publications with a 3.20 median impact factor. Only 107 were available online. Most primary authors originated from USA, UK and the Netherlands. The top three publication types were cohort studies, surveys and clinical trials. Regarding indicators of reproducibility, 17 publications gave access to necessary materials, four provided in-depth protocol and one contained raw data required to reproduce the outcomes. One publication offered its analysis script on request; four provided a protocol availability statement. Only 107 publications were publicly available: 13 were registered in online repositories and four, ten and eight publications included their hypothesis, methods and analysis, respectively. Conflict of interest was addressed by 177 and reported by 31 publications. Of 185 publications with a funding statement, 153 publications were funded and 32 were unfunded.

CONCLUSIONS

Currently, Psychiatry research has significant potential to improve adherence to reproducibility and transparency practices. Thus, this study presents a reference point for the state of reproducibility and transparency in Psychiatry literature. Future assessments are recommended to evaluate and encourage progress.

Selective reporting bias due to discrepancies between registered and published outcomes in osteoarthritis trials

Aim: Outcome reporting bias (ORB) occurs when outcomes planned in a study protocol are subsequently not reported or are partially reported. Our aim was to analyze ORB in randomized controlled trials (RCTs) about conservative interventions for osteoarthritis (OA) by comparing registered protocols and published manuscripts, as well as association between study funding type and intervention type, and ORB in those RCTs. Materials & methods: We analyzed RCTs that were published in a peer-review journal and analyzed any type of conservative intervention for treatment of OA in humans that reported in the manuscript registration in a public clinical trial registry and provided unique registration identifier. We extracted data indicating ORB by comparing outcomes in protocol and published article, and characteristics of trials. Results: In 190 (57%) of 334 included RCTs, it was indicated in the manuscript that a trial was registered. In 48% of trials we found discrepancies in number, type or time point of primary efficacy outcome between protocol and manuscript. Significantly less discrepancies in primary efficacy outcomes between protocols and published articles were found in trials funded by a commercial sponsor (p = 0.0062) and trials of pharmacological interventions (p = 0.0016). Conclusion: Trials about conservative therapies for OA have high prevalence of discrepancies between protocol and publication, and frequent ORB. This may mislead readers of published results because it has been shown that ORB can lead to both overestimation and underestimation of effects of interventions, depending on the intervention and outcome. Efforts to prevent nonregistration of protocols and selective reporting are needed.

Discrepancies between Registered and Published Primary and Secondary Outcomes in Randomized Controlled Trials within the Plastic Surgery Literature: A Systematic Review

BACKGROUND

Recent studies have identified a high incidence of discrepancy between registered and published outcomes in registered medical and surgical randomized controlled trials. This has not yet been studied in the plastic surgery literature.

METHODS

The authors systematically assessed plastic surgery randomized controlled trials published between 2012 and 2016 in seven high-impact plastic surgery journals. Data were collected from the registration website and published articles using a standardized data extraction form.

RESULTS

A total of 145 randomized controlled trials were identified, with a 39 percent trial registration rate (n = 57). Forty-nine trials were included in the final analysis. Forty-three (88 percent) had a discrepancy between registered and published outcomes: 26 (53 percent) for primary outcome(s), and 39 (80 percent) for secondary outcome(s). The number of discrepancies in an individual trial ranged from one to seven for primary outcomes and one to 12 for secondary outcomes. Aesthetic surgery had the largest number of trials with outcome discrepancies (n = 15). The prevalence of unreported registered outcomes was 13 percent for primary outcomes and 38 percent for secondary outcomes. Registered nonsignificant primary outcomes were published as nonsignificant secondary outcomes in 30 percent of trials. Publishing new nonregistered secondary outcomes (65 percent) and changing the assessment timing of published primary outcomes (61 percent) were the most common types of discrepancies. Discrepancies favored a statistically significant positive outcome in 19 (44 percent) of the 43 trials with an outcome discrepancy. Discrepancies that resulted in published outcomes with improved patient relevance were found in eight trials (16 percent) for primary outcome discrepancies and 14 trials (29 percent) for secondary outcome discrepancies.

CONCLUSIONS

The plastic surgery literature has high rates of discrepancies between registered and published trial outcomes. Outcome reporting discrepancy is even more problematic for secondary outcomes, an area of analysis that has previously been poorly studied. The high rate of discrepancy change favoring a statistically significant outcome and more patient-relevant outcomes may indicate the pressure to demonstrate significant results to be accepted for publication in high-impact journals.

Increase value and reduce waste in research on psychological therapies

A seminal Lancet series focused on increasing value and reducing waste in biomedical research, providing a transferrable template to diagnose problems in research. Our goal was to document how some of these sources of waste apply to mental health and particularly psychological treatments research. We synthesize and critically evaluate empirical findings in relation to four major sources: i) defining research priorities; ii) research design, methods and analysis; iii) accessibility of research information; iv) accuracy and usability of research reports. We demonstrate that each source of waste considered is well-represented and amply documented within this field. We describe hype and insufficient consideration of what is known in defining research priorities, persistent risk of bias, particularly due to selective outcome reporting, for psychotherapy trials across mental disorders, intellectual and financial biases, direct and indirect evidence of publication bias, largely inexistent adoption of data sharing, issues of multiplicity and fragmentation of data and findings, and insufficient adoption of reporting guidelines. We expand on a few general solutions, including supporting meta-research, properly testing interventions to increase research quality, placing open science at the center of psychological treatment research and remaining vigilant particularly regarding the strains of research currently prioritized, such as experimental psychopathology.

Evaluation of Journal Registration Policies and Prospective Registration of Randomized Clinical Trials of Nonregulated Health Care Interventions

IMPORTANCE

Many interventions that are important to the health care of patients are not subject to regulation by the US Food and Drug Administration (FDA) or comparable regulatory bodies in other nations.

OBJECTIVE

To determine whether specialty journals that publish trials of primarily nonregulated health care interventions require prospective registration and whether the prospective registration policies are associated with the publication of prospectively registered trials, trials with adequately registered outcomes, and trials with primary outcomes consistent with the registered primary outcomes.

DESIGN AND METHODS

PubMed was searched daily, from March 18, 2016, to September 17, 2016, for nonregulated intervention randomized clinical trials. The search included all journals in the Clarivate Analytics Science Citation Index Expanded categories of behavioral sciences, nursing, nutrition and dietetics, psychology, rehabilitation, and surgery. Trials of interventions not subject to FDA regulation were included. One investigator extracted journal registration policy and trial registration status. Two investigators independently extracted trial registration and publication characteristics.

MAIN OUTCOMES AND MEASURES

For journals, the main outcome was the trial registration policy. For trials, the main outcomes were prospective registration, adequacy of outcome registration, and concordance of registered with published primary outcomes.

RESULTS

In total, 953 nonregulated intervention trials published in 254 journals were identified. Prospective registration was required for publication by 29 (11.4%) of 254 journals, and an additional 12 journals (4.7%) had conditional date-based requirements. Only 189 (19.8%) of the 953 trials were registered prospectively, including 33 of 98 published in journals with prospective registration policies as compared with 156 of 855 in journals without policies (33.7% vs 18.2%; P = .004). Among the 17 journals that required prospective registration and had at least 2 included trials, none had a prospective registration of more than 50%. In journals with policies, only 3 of 98 trials included primary outcomes consistent with prospectively, adequately registered outcomes, as compared with 34 of 852 trials in journals without policies (3.1% vs 4.0%; P = .62).

CONCLUSIONS AND RELEVANCE

Few journals in behavioral sciences or psychology, nursing, nutrition and dietetics, rehabilitation, and surgery require prospective trial registration, and those with existing registration policies rarely enforce them; this finding suggests that strategies for encouraging prospective registration of clinical trials not subject to FDA regulation should be developed and tested.

Registration of published randomized trials: a systematic review and meta-analysis

BACKGROUND

Prospective trial registration is a powerful tool to prevent reporting bias. We aimed to determine the extent to which published randomized controlled trials (RCTs) were registered and registered prospectively.

METHODS

We searched MEDLINE and EMBASE from January 2005 to October 2017; we also screened all articles cited by or citing included and excluded studies, and the reference lists of related reviews. We included studies that examined published RCTs and evaluated their registration status, regardless of medical specialty or language. We excluded studies that assessed RCT registration status only through mention of registration in the published RCT, without searching registries or contacting the trial investigators. Two independent reviewers blinded to the other's work performed the selection. Following PRISMA guidelines, two investigators independently extracted data, with discrepancies resolved by consensus. We calculated pooled proportions and 95% confidence intervals using random-effects models.

RESULTS

We analyzed 40 studies examining 8773 RCTs across a wide range of clinical specialties. The pooled proportion of registered RCTs was 53% (95% confidence interval 44% to 58%), with considerable between-study heterogeneity. A subset of 24 studies reported data on prospective registration across 5529 RCTs. The pooled proportion of prospectively registered RCTs was 20% (95% confidence interval 15% to 25%). Subgroup analyses showed that registration was higher for industry-supported and larger RCTs. A meta-regression analysis across 19 studies (5144 RCTs) showed that the proportion of registered trials significantly increased over time, with a mean proportion increase of 27%, from 25 to 52%, between 2005 and 2015.

CONCLUSIONS

The prevalence of trial registration has increased over time, but only one in five published RCTs is prospectively registered, undermining the validity and integrity of biomedical research.

An Evaluation of Reporting Guidelines and Clinical Trial Registry Requirements Among Orthopaedic Surgery Journals

BACKGROUND

The responsibility for ensuring that studies are adequately reported is primarily that of those conducting the study; however, journal policies may influence how thoroughly authors choose to report their research. The use of reporting guidelines and prospective trial registration are promising avenues for ensuring that published studies adhere to the highest methodological standards. The purpose of this study is to evaluate orthopaedic surgery journal policies regarding reporting guidelines and trial registration, and to evaluate the effects that these policies have on adherence to reporting.

METHODS

We conducted a cross-sectional survey of journal policies and "Instructions for Authors" to determine the journals' policies and guidance regarding use of reporting guidelines and study registration. We also examined whether trials published in journals referencing CONSORT (Consolidated Standards of Reporting Trials) had higher rates of compliance with publishing a CONSORT flow diagram and whether journals with trial registration policies were more likely to contain registered trials than journals without these requirements.

RESULTS

Of the 21 orthopaedic surgery journals, 6 (29%) did not mention a single guideline, and clinical trial registration was required by 11 (52%) orthopaedic surgery journals and recommended by 2 (10%). Of the 21 general medical journals, 3 (14%) did not mention a single guideline, and trial registration was required by 13 (62%) general medical journals and recommended by 5 (24%) others. Furthermore, journals that referenced CONSORT were more likely to publish trials with a CONSORT flow diagram. Journals with trial registration policies were more likely to publish registered trials.

CONCLUSIONS

Reporting guidelines and trial registration are suboptimally required or recommended by orthopaedic surgery journals. These 2 mechanisms may improve methodology and quality, and should be considered for adoption by journal editors in orthopaedic surgery.

CLINICAL RELEVANCE

Because orthopaedic surgeons rely on high-quality research to direct patient care, measures must be taken to ensure that published research is of the highest quality. The use of reporting guidelines and prospective clinical trial registration may improve the quality of orthopaedic research, thereby improving patient care.

The limitations of using randomised controlled trials as a basis for developing treatment guidelines

Randomised controlled trials (RCTs) are considered the 'gold standard' by which novel psychotropic medications and psychological interventions are evaluated and consequently adopted into widespread clinical practice. However, there are some limitations to using RCTs as the basis for developing treatment guidelines. While RCTs allow researchers to determine whether a given medication or intervention is effective in a specific patient sample, for practicing clinicians it is more important to know whether it will work for their particular patient in their particular setting. This information cannot be garnered from an RCT. These inherent limitations are exacerbated by biases in design, recruitment, sample populations and data analysis that are inevitable in real-world studies. While trial registration and CONSORT have been implemented to correct and improve these issues, it is worrying that many trials fail to achieve such standards and yet their findings are used to inform clinical decision making. This perspective piece questions the assumptions of RCTs and highlights the widespread distortion of findings that currently undermine the credibility of this powerful design. It is recommended that the clinical guidelines include advice as to what should be considered good and relevant evidence and that external bodies continue to monitor RCTs to ensure that the outcomes published indeed reflect reality.

Published intimate partner violence studies often differ from their trial registration records

INTRODUCTION

Registering study protocols in a trial registry is important for methodologic transparency and reducing selective reporting bias. The objective of this investigation was to determine whether published studies of intimate partner violence (IPV) that had been registered matched the registration record on key study design elements.

METHODS

We systematically searched three trial registries to identify registered IPV studies and the published literature for the associated publication. Two authors independently determined for each study whether key study elements in the registry matched those in the published paper.

RESULTS

We included 66 studies published between 2006 and 2017. Nearly half (29/66, 44%) were registered after study completion. Many (26/66, 39%) had discrepancies regarding the primary outcome, and nearly two-thirds (42/66, 64%) had discrepancies in secondary outcomes. Discrepancies in study design were less frequent (13/66, 20%). However, large changes in sample size (26/66, 39%) and discrepancies in funding source (28/66, 42%) were frequently observed.

CONCLUSIONS

Trial registries are important tools for research transparency and identifying and preventing outcome switching and selective outcome reporting bias. Published IPV studies often differ from their records in trial registries. Researchers should pay close attention to the accuracy of trial registry records.

[Transparency and replicability of nursing intervention studies in long-term care: A selective literature review]

BACKGROUND AND OBJECTIVE

The development and evaluation of interventions in long-term care is time-consuming and expensive due to their complexity. To ensure reproducibility and successful implementation, these interventions must be described and published in a comprehensible and qualitative manner. The aim of this study is to analyze intervention studies from the inpatient long-term care setting with regard to their completeness, reporting quality, transparency and thus reproducibility.

METHOD

The completeness and the reporting quality of the interventions described in the publications were examined in the context of a selective literature review by means of intervention studies from the long-term care setting (n=22). To this end, the Template for Intervention Description and Replication (TIDieR) checklist and the Criteria for Reporting the Development and Evaluation of Complex Interventions in Healthcare 2 (CReDECI2-DE) list were used. Transparency criteria included study registration and access to study protocols.

RESULTS

The TIDieR checklist examination revealed that only three studies contained all the information necessary; the CReDECI2 test provided a complete description for only one study. Frequent shortcomings were observed concerning the information on modifications and titrations for the study participants and the location. Protocols were available for eight studies, 14 studies were registered.

CONCLUSIONS

In terms of science, this means that the reproducibility of scientific findings is limited, which is why they cannot provide secure knowledge. As a result, the practical benefit to be derived from published studies that are accessible to decision-makers is limited as well. As far as publishers are concerned they should pay more attention to the completeness, registration and availability of materials.

Common versus specific factors in psychotherapy: opening the black box

Do psychotherapies work primarily through the specific factors described in treatment manuals, or do they work through common factors? In attempting to unpack this ongoing debate between specific and common factors, we highlight limitations in the existing evidence base and the power battles and competing paradigms that influence the literature. The dichotomy is much less than it might first appear. Most specific factor theorists now concede that common factors have importance, whereas the common factor theorists produce increasingly tight definitions of bona fide therapy. Although specific factors might have been overplayed in psychotherapy research, some are effective for particular conditions. We argue that continuing to espouse common factors with little evidence or endless head-to-head comparative studies of different psychotherapies will not move the field forward. Rather than continuing the debate, research needs to encompass new psychotherapies such as e-therapies, transdiagnostic treatments, psychotherapy component studies, and findings from neurobiology to elucidate the effective process components of psychotherapy.

Component studies of psychological treatments of adult depression: A systematic review and meta-analysis

OBJECTIVES

A recent report from the US Institute of Medicine indicated that identifying core elements of psychosocial interventions is a key step in successfully bringing evidence-based psychosocial interventions into clinical practice. Component studies have the best design to examine these core elements. Earlier reviews resulted in heterogeneous sets of studies and probably missed many studies.

METHODS

We conducted a comprehensive search of component studies on psychotherapies for adult depression and included 16 studies with 22 comparisons.

RESULTS

Fifteen components were examined of which four were examined in more than one comparison. The pooled difference between the full treatments and treatments with one component removed was g = 0.21 (95% CI: 0.03∼0.39). One study had sufficient statistical power to detect a small effect size and found that adding emotion regulation skills increased the effects of CBT. None of the other studies had enough power to detect an effect size smaller than g = 0.55. Only one study had low risk of bias.

CONCLUSIONS

The currently available component studies do not have the statistical power nor the quality to draw any meaningful conclusion about key ingredients of psychotherapies for adult depression.

Does psychotherapy work? An umbrella review of meta-analyses of randomized controlled trials

OBJECTIVE

To map and evaluate the evidence across meta-analyses of randomized controlled trials (RCTs) of psychotherapies for various outcomes.

METHODS

We identified 173 eligible studies, including 247 meta-analyses that synthesized data from 5157 RCTs via a systematic search from inception to December 2016 in the PubMed, PsycINFO and Cochrane Database of Systematic Reviews. We calculated summary effects using random-effects models, and we assessed between-study heterogeneity. We estimated whether large studies had significantly more conservative results compared to smaller studies (small-study effects) and whether the observed positive studies were more than expected by chance. Finally, we assessed the credibility of the evidence using several criteria.

RESULTS

One hundred and ninety-nine meta-analyses were significant at P-value ≤ 0.05, and almost all (n = 196) favoured psychotherapy. Large and very large heterogeneity was observed in 130 meta-analyses. Evidence for small-study effects was found in 72 meta-analyses, while 95 had evidence of excess of significant findings. Only 16 (7%) provided convincing evidence that psychotherapy is effective. These pertained to cognitive behavioural therapy (n = 6), meditation therapy (n = 1), cognitive remediation (n = 1), counselling (n = 1) and mixed types of psychotherapies (n = 7).

CONCLUSIONS

Although almost 80% meta-analyses reported a nominally statistically significant finding favouring psychotherapy, only a few meta-analyses provided convincing evidence without biases.

Why prudence is needed when interpreting articles reporting clinical trial results in mental health

Background

Clinical trial results’ reliability is impacted by reporting bias. This is primarily manifested as publication bias and outcome reporting bias.

Mental health trials’ specific features

Mental health trials are prone to two methodological deficiencies: (1) using small numbers of participants that facilitates false positive findings and exaggerated size effects, and (2) the obligatory use of psychometric scales that require subjective assessments. These two deficiencies contribute to the publication of unreliable results. Considerable reporting bias has been found in safety and efficacy findings in psychotherapy and pharmacotherapy trials. Reporting bias can be carried forward to meta-analyses, a key source for clinical practice guidelines. The final result is the frequent overestimation of treatment effects that could impact patients and clinician-informed decisions.

Mechanisms to prevent outcome reporting bias

Prospective registration of trials and publication of results are the two major methods to reduce reporting bias. Prospective trial registration will allow checking whether they are published (so it will help to prevent publication bias) and, if published, whether those outcomes and analyses that were deemed as appropriate before trial commencement are actually published (hence helping to find out selective reporting of outcomes). Unfortunately, the rate of registered trials in mental health interventions is low and, frequently, of poor quality.

Conclusion

Clinicians should be prudent when interpreting the results of published trials and some meta-analyses – such as those conducted by scientists working for the sponsor company or those that only include published trials. Prescribers, however, should be confident when prescribing drugs following the summary of product characteristics, since regulatory agencies have access to all clinical trial results.