A guide for social science journal editors on easing into open science

Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org ) has collated several resources on embedding open science in journal editing ( www.dpjedi.org/resources ). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx ).

Publication of Results of Registered Trials With Published Study Protocols, 2011-2022

Importance

Publishing study protocols might reduce research waste because of unclear methods or incomplete reporting; on the other hand, there might be few additional benefits of publishing protocols for registered trials that are never completed or published. No study has investigated the proportion of published protocols associated with published results.

Objective

To estimate the proportion of published trial protocols for which there are not associated published results.

Design, Setting, and Participants

This cross-sectional study used stratified random sampling to identify registered clinical trials with protocols published between January 2011 and August 2022 and indexed in PubMed Central. Ongoing studies and those within 1 year of the primary completion date on ClinicalTrials.gov were excluded. Published results were sought from August 2022 to March 2023 by searching ClinicalTrials.gov, emailing authors, and using an automated tool, as well as through incidental discovery.

Main Outcomes and Measures

The primary outcome was a weighted estimate of the proportion of registered trials with published protocols that also had published main results. The proportion of trials with unpublished results was estimated using a weighted mean.

Results

From 1500 citations that were screened, 308 clinical trial protocols were included, and it was found that 87 trials had not published their main results. Most included trials were investigator-initiated evaluations of nonregulated products. When published, results appeared a mean (SD) of 3.4 (2.0) years after protocol publications. With the use of a weighted mean, an estimated 4754 (95% CI, 4296-5226) eligible clinical trial protocols were published and indexed in PubMed Central between 2011 and 2022. In the weighted analysis, 1708 of those protocols (36%; 95% CI, 31%-41%) were not associated with publication of main results. In a sensitivity analysis excluding protocols published after 2019, an estimated 25% (95% CI, 20%-30%) of 3670 (95% CI, 3310-4032) protocol publications were not associated with publication of main results.

Conclusions and Relevance

This cross-sectional study of clinical trial protocols published on PubMed Central between 2011 and 2022 suggests that many protocols were not associated with subsequent publication of results. The overall benefits of publishing study protocols might outweigh the research waste caused by unnecessary protocol publications.

Harms were detected but not reported in six clinical trials of gabapentin

OBJECTIVES

We sought to assess and report harms that were observed but not disclosed previously in clinical trials of gabapentin.

STUDY DESIGN AND SETTING

We reanalyzed individual participant data from six randomized parallel trials of gabapentin for neuropathic pain, and we conducted meta-analyses. Between 1996 and 2003, adult participants were assigned to gabapentin (600 mg-3,600 mg per day) or placebo for 7-14 weeks. We calculated the proportion of participants with: one or more adverse events (AEs), one or more serious AEs, discontinued, discontinued because AEs. We also estimated effects on AEs at three levels of aggregation using COSTART, a hierarchical system for classifying AEs: body system, midlevel system, preferred term.

RESULTS

We found evidence of important harms that were neither included in published trial reports nor included in systematic reviews. Aggregating related harms led to greater confidence that gabapentin might harm the nervous system and possibly the digestive, metabolic and nutritional, respiratory, sensory, and urogenital body systems. Nervous system harms were more common than previous reports suggest.

CONCLUSION

Clinical trials identified harms that were not reported publicly, and journal articles overstated uncertainty about harms. Relying on journal articles to evaluate gabapentin's harms could contribute to incomplete and misleading conclusions in systematic reviews and guidelines. To prevent bias arising from the selective nonreporting of results, journal articles should describe how to access data for all harms observed in clinical trials (e.g., by sharing individual participant data).

Open Science Standards at Journals that Inform Evidence-Based Policy

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

Methodology reporting improved over time in 176,469 randomized controlled trials

OBJECTIVES

To describe randomized controlled trial (RCT) methodology reporting over time.

STUDY DESIGN AND SETTING

We used a deep learning-based sentence classification model based on the Consolidated Standards of Reporting Trials (CONSORT) statement, considered minimum requirements for reporting RCTs. We included 176,469 RCT reports published between 1966 and 2018. We analyzed the reporting trends over 5-year time periods, grouping trials from 1966 to 1990 in a single stratum. We also explored the effect of journal impact factor (JIF) and medical discipline.

RESULTS

Population, Intervention, Comparator, Outcome (PICO) items were commonly reported during each period, and reporting increased over time (e.g., interventions: 79.1% during 1966-1990 to 87.5% during 2010-2018). Reporting of some methods information has increased, although there is room for improvement (e.g., sequence generation: 10.8-41.8%). Some items are reported infrequently (e.g., allocation concealment: 5.1-19.3%). The number of items reported and JIF are weakly correlated (Pearson's r (162,702) = 0.16, P < 0.001). The differences in the proportion of items reported between disciplines are small (<10%).

CONCLUSION

Our analysis provides large-scale quantitative support for the hypothesis that RCT methodology reporting has improved over time. Extending these models to all CONSORT items could facilitate compliance checking during manuscript authoring and peer review, and support metaresearch.

CONSORT Harms 2022 statement, explanation, and elaboration: updated guideline for the reporting of harms in randomized trials

Randomized controlled trials remain the reference standard for healthcare research on effects of interventions, and the need to report both benefits and harms is essential. The Consolidated Standards of Reporting Trials (the main CONSORT) statement includes one item on reporting harms (i.e., all important harms or unintended effects in each group). In 2004, the CONSORT group developed the CONSORT Harms extension; however, it has not been consistently applied and needs to be updated. Here, we describe CONSORT Harms 2022, which replaces the CONSORT Harms 2004 checklist, and shows how CONSORT Harms 2022 items could be incorporated into the main CONSORT checklist. Thirteen items from the main CONSORT were modified to improve harms reporting. Three new items were added. In this article, we describe CONSORT Harms 2022 and how it was integrated into the main CONSORT checklist and elaborate on each item relevant to complete reporting of harms in randomized controlled trials. Until future work from the CONSORT group produces an updated checklist, authors, journal reviewers, and editors of randomized controlled trials should use the integrated checklist presented in this paper.

Conducting separate reviews of benefits and harms could improve systematic reviews and meta-analyses

Guidance for systematic reviews of interventions recommends both benefits and harms be included. Systematic reviews may reach conclusions about harms (or lack of harms) that are not true when reviews include only some relevant studies, rely on incomplete data from eligible studies, use inappropriate methods for synthesizing data, and report results selectively. Separate reviews about harms could address some of these problems, and we argue that conducting separate reviews of harms is a feasible alternative to current standards and practices. Systematic reviews of potential benefits could be organized around the use of interventions for specific health problems. Systematic reviews of potential harms could be broader, including more diverse study designs and including all people at risk of harms (who might use the same intervention to treat different health problems). Multiple reviews about benefits could refer to a single review of harms. This approach could improve the reliability, completeness, and efficiency of systematic reviews.

Zinc supplementation for preventing mortality, morbidity, and growth failure in children aged 6 months to 12 years

BACKGROUND

Zinc deficiency is prevalent in low- and middle-income countries, and is considered a significant risk factor for morbidity, mortality, and linear growth failure. The effectiveness of preventive zinc supplementation in reducing prevalence of zinc deficiency needs to be assessed.

OBJECTIVES

To assess the effects of zinc supplementation for preventing mortality and morbidity, and for promoting growth, in children aged 6 months to 12 years.

SEARCH METHODS

A previous version of this review was published in 2014. In this update, we searched CENTRAL, MEDLINE, Embase, five other databases, and one trials register up to February 2022, together with reference checking and contact with study authors to identify additional studies.

SELECTION CRITERIA

Randomized controlled trials (RCTs) of preventive zinc supplementation in children aged 6 months to 12 years compared with no intervention, a placebo, or a waiting list control. We excluded hospitalized children and children with chronic diseases or conditions. We excluded food fortification or intake, sprinkles, and therapeutic interventions.

DATA COLLECTION AND ANALYSIS

Two review authors screened studies, extracted data, and assessed the risk of bias. We contacted study authors for missing information and used GRADE to assess the certainty of evidence. The primary outcomes of this review were all-cause mortality; and cause-specific mortality, due to all-cause diarrhea, lower respiratory tract infection (LRTI, including pneumonia), and malaria. We also collected information on a number of secondary outcomes, such as those related to diarrhea and LRTI morbidity, growth outcomes and serum levels of micronutrients, and adverse events.

MAIN RESULTS

We included 16 new studies in this review, resulting in a total of 96 RCTs with 219,584 eligible participants. The included studies were conducted in 34 countries; 87 of them in low- or middle-income countries. Most of the children included in this review were under five years of age. The intervention was delivered most commonly in the form of syrup as zinc sulfate, and the most common dose was between 10 mg and 15 mg daily. The median duration of follow-up was 26 weeks. We did not consider that the evidence for the key analyses of morbidity and mortality outcomes was affected by risk of bias. High-certainty evidence showed little to no difference in all-cause mortality with preventive zinc supplementation compared to no zinc (risk ratio (RR) 0.93, 95% confidence interval (CI) 0.84 to 1.03; 16 studies, 17 comparisons, 143,474 participants). Moderate-certainty evidence showed that preventive zinc supplementation compared to no zinc likely results in little to no difference in mortality due to all-cause diarrhea (RR 0.95, 95% CI 0.69 to 1.31; 4 studies, 132,321 participants); but probably reduces mortality due to LRTI (RR 0.86, 95% CI 0.64 to 1.15; 3 studies, 132,063 participants) and mortality due to malaria (RR 0.90, 95% CI 0.77 to 1.06; 2 studies, 42,818 participants); however, the confidence intervals around the summary estimates for these outcomes were wide, and we could not rule out a possibility of increased risk of mortality. Preventive zinc supplementation likely reduces the incidence of all-cause diarrhea (RR 0.91, 95% CI 0.90 to 0.93; 39 studies, 19,468 participants; moderate-certainty evidence) but results in little to no difference in morbidity due to LRTI (RR 1.01, 95% CI 0.95 to 1.08; 19 studies, 10,555 participants; high-certainty evidence) compared to no zinc. There was moderate-certainty evidence that preventive zinc supplementation likely leads to a slight increase in height (standardized mean difference (SMD) 0.12, 95% CI 0.09 to 0.14; 74 studies, 20,720 participants). Zinc supplementation was associated with an increase in the number of participants with at least one vomiting episode (RR 1.29, 95% CI 1.14 to 1.46; 5 studies, 35,192 participants; high-certainty evidence). We report a number of other outcomes, including the effect of zinc supplementation on weight and serum markers such as zinc, hemoglobin, iron, copper, etc. We also performed a number of subgroup analyses and there was a consistent finding for a number of outcomes that co-supplementation of zinc with iron decreased the beneficial effect of zinc.

AUTHORS' CONCLUSIONS

Even though we included 16 new studies in this update, the overall conclusions of the review remain unchanged. Zinc supplementation might help prevent episodes of diarrhea and improve growth slightly, particularly in children aged 6 months to 12 years of age. The benefits of preventive zinc supplementation may outweigh the harms in regions where the risk of zinc deficiency is relatively high.

Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors

The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.

Evaluating Implementation of the Transparency and Openness Promotion Guidelines: Reliability of Instruments to Assess Journal Policies, Procedures, and Practices

The Transparency and Openness Promotion (TOP) Guidelines describe modular standards that journals can adopt to promote open science. The TOP Factor quantifies the extent to which journals adopt TOP in their policies, but there is no validated instrument to assess TOP implementation. Moreover, raters might assess the same policies differently. Instruments with objective questions are needed to assess TOP implementation reliably. In this study, we examined the interrater reliability and agreement of three new instruments for assessing TOP implementation in journal policies (instructions to authors), procedures (manuscript-submission systems), and practices (journal articles). Independent raters used these instruments to assess 339 journals from the behavioral, social, and health sciences. We calculated interrater agreement (IRA) and interrater reliability (IRR) for each of 10 TOP standards and for each question in our instruments (13 policy questions, 26 procedure questions, 14 practice questions). IRA was high for each standard in TOP; however, IRA might have been high by chance because most standards were not implemented by most journals. No standard had “excellent” IRR. Three standards had “good,” one had “moderate,” and six had “poor” IRR. Likewise, IRA was high for most instrument questions, and IRR was moderate or worse for 62%, 54%, and 43% of policy, procedure, and practice questions, respectively. Although results might be explained by limitations in our process, instruments, and team, we are unaware of better methods for assessing TOP implementation. Clarifying distinctions among different levels of implementation for each TOP standard might improve its implementation and assessment (study protocol: https://doi.org/10.1186/s41073-021-00112-8 ).

[The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsDeclaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas]

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.

Guidelines for Reporting Outcomes in Trial Protocols: The SPIRIT-Outcomes 2022 Extension

IMPORTANCE

Complete information in a trial protocol regarding study outcomes is crucial for obtaining regulatory approvals, ensuring standardized trial conduct, reducing research waste, and providing transparency of methods to facilitate trial replication, critical appraisal, accurate reporting and interpretation of trial results, and knowledge synthesis. However, recommendations on what outcome-specific information should be included are diverse and inconsistent. To improve reporting practices promoting transparent and reproducible outcome selection, assessment, and analysis, a need for specific and harmonized guidance as to what outcome-specific information should be addressed in clinical trial protocols exists.

OBJECTIVE

To develop harmonized, evidence- and consensus-based standards for describing outcomes in clinical trial protocols through integration with the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 statement.

EVIDENCE REVIEW

Using the Enhancing the Quality and Transparency of Health Research (EQUATOR) methodological framework, the SPIRIT-Outcomes 2022 extension of the SPIRIT 2013 statement was developed by (1) generation and evaluation of candidate outcome reporting items via consultation with experts and a scoping review of existing guidance for reporting trial outcomes (published within the 10 years prior to March 19, 2018) identified through expert solicitation, electronic database searches of MEDLINE and the Cochrane Methodology Register, gray literature searches, and reference list searches; (2) a 3-round international Delphi voting process (November 2018-February 2019) completed by 124 panelists from 22 countries to rate and identify additional items; and (3) an in-person consensus meeting (April 9-10, 2019) attended by 25 panelists to identify essential items for outcome-specific reporting to be addressed in clinical trial protocols.

FINDINGS

The scoping review and consultation with experts identified 108 recommendations relevant to outcome-specific reporting to be addressed in trial protocols, the majority (72%) of which were not included in the SPIRIT 2013 statement. All recommendations were consolidated into 56 items for Delphi voting; after the Delphi survey process, 19 items met criteria for further evaluation at the consensus meeting and possible inclusion in the SPIRIT-Outcomes 2022 extension. The discussions during and after the consensus meeting yielded 9 items that elaborate on the SPIRIT 2013 statement checklist items and are related to completely defining and justifying the choice of primary, secondary, and other outcomes (SPIRIT 2013 statement checklist item 12) prospectively in the trial protocol, defining and justifying the target difference between treatment groups for the primary outcome used in the sample size calculations (SPIRIT 2013 statement checklist item 14), describing the responsiveness of the study instruments used to assess the outcome and providing details on the outcome assessors (SPIRIT 2013 statement checklist item 18a), and describing any planned methods to account for multiplicity relating to the analyses or interpretation of the results (SPIRIT 2013 statement checklist item 20a).

CONCLUSIONS AND RELEVANCE

This SPIRIT-Outcomes 2022 extension of the SPIRIT 2013 statement provides 9 outcome-specific items that should be addressed in all trial protocols and may help increase trial utility, replicability, and transparency and may minimize the risk of selective nonreporting of trial results.

Guidelines for Reporting Outcomes in Trial Reports: The CONSORT-Outcomes 2022 Extension

IMPORTANCE

Clinicians, patients, and policy makers rely on published results from clinical trials to help make evidence-informed decisions. To critically evaluate and use trial results, readers require complete and transparent information regarding what was planned, done, and found. Specific and harmonized guidance as to what outcome-specific information should be reported in publications of clinical trials is needed to reduce deficient reporting practices that obscure issues with outcome selection, assessment, and analysis.

OBJECTIVE

To develop harmonized, evidence- and consensus-based standards for reporting outcomes in clinical trial reports through integration with the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement.

EVIDENCE REVIEW

Using the Enhancing the Quality and Transparency of Health Research (EQUATOR) methodological framework, the CONSORT-Outcomes 2022 extension of the CONSORT 2010 statement was developed by (1) generation and evaluation of candidate outcome reporting items via consultation with experts and a scoping review of existing guidance for reporting trial outcomes (published within the 10 years prior to March 19, 2018) identified through expert solicitation, electronic database searches of MEDLINE and the Cochrane Methodology Register, gray literature searches, and reference list searches; (2) a 3-round international Delphi voting process (November 2018-February 2019) completed by 124 panelists from 22 countries to rate and identify additional items; and (3) an in-person consensus meeting (April 9-10, 2019) attended by 25 panelists to identify essential items for the reporting of outcomes in clinical trial reports.

FINDINGS

The scoping review and consultation with experts identified 128 recommendations relevant to reporting outcomes in trial reports, the majority (83%) of which were not included in the CONSORT 2010 statement. All recommendations were consolidated into 64 items for Delphi voting; after the Delphi survey process, 30 items met criteria for further evaluation at the consensus meeting and possible inclusion in the CONSORT-Outcomes 2022 extension. The discussions during and after the consensus meeting yielded 17 items that elaborate on the CONSORT 2010 statement checklist items and are related to completely defining and justifying the trial outcomes, including how and when they were assessed (CONSORT 2010 statement checklist item 6a), defining and justifying the target difference between treatment groups during sample size calculations (CONSORT 2010 statement checklist item 7a), describing the statistical methods used to compare groups for the primary and secondary outcomes (CONSORT 2010 statement checklist item 12a), and describing the prespecified analyses and any outcome analyses not prespecified (CONSORT 2010 statement checklist item 18).

CONCLUSIONS AND RELEVANCE

This CONSORT-Outcomes 2022 extension of the CONSORT 2010 statement provides 17 outcome-specific items that should be addressed in all published clinical trial reports and may help increase trial utility, replicability, and transparency and may minimize the risk of selective nonreporting of trial results.

Feasibility of an Audit and Feedback Intervention to Facilitate Journal Policy Change Towards Greater Promotion of Transparency and Openness in Sports Science Research

Objectives

To evaluate (1) the feasibility of an audit-feedback intervention to facilitate sports science journal policy change, (2) the reliability of the Transparency of Research Underpinning Social Intervention Tiers (TRUST) policy evaluation form, and (3) the extent to which policies of sports science journals support transparent and open research practices.

Methods

We conducted a cross-sectional, audit-feedback, feasibility study of transparency and openness standards of the top 38 sports science journals by impact factor. The TRUST form was used to evaluate journal policies support for transparent and open research practices. Feedback was provided to journal editors in the format of a tailored letter. Inter-rater reliability and agreement of the TRUST form was assessed using intraclass correlation coefficients and the standard error of measurement, respectively. Time-based criteria, fidelity of intervention delivery and qualitative feedback were used to determine feasibility.

Results

The audit-feedback intervention was feasible based on the time taken to rate journals and provide tailored feedback. The mean (SD) score on the TRUST form (range 0–27) was 2.05 (1.99), reflecting low engagement with transparent and open practices. Inter-rater reliability of the overall score of the TRUST form was moderate [ICC (2,1) = 0.68 (95% CI 0.55–0.79)], with standard error of measurement of 1.17. However, some individual items had poor reliability.

Conclusion

Policies of the top 38 sports science journals have potential for improved support for transparent and open research practices. The feasible audit-feedback intervention developed here warrants large-scale evaluation as a means to facilitate change in journal policies.

Registration: OSF (https://osf.io/d2t4s/).

Supplementary Information

The online version contains supplementary material available at 10.1186/s40798-022-00496-x.

Comparing the Value of Data Visualization Methods for Communicating Harms in Clinical Trials

In clinical trials, harms (i.e., adverse events) are often reported by simply counting the number of people who experienced each event. Reporting only frequencies ignores other dimensions of the data that are important for stakeholders, including severity, seriousness, rate (recurrence), timing, and groups of related harms. Additionally, application of selection criteria to harms prevents most from being reported. Visualization of data could improve communication of multidimensional data. We replicated and compared the characteristics of 6 different approaches for visualizing harms: dot plot, stacked bar chart, volcano plot, heat map, treemap, and tendril plot. We considered binary events using individual participant data from a randomized trial of gabapentin for neuropathic pain. We assessed their value using a heuristic approach and a group of content experts. We produced all figures using R and share the open-source code on GitHub. Most original visualizations propose presenting individual harms (e.g., dizziness, somnolence) alone or alongside higher level (e.g., by body systems) summaries of harms, although they could be applied at either level. Visualizations can present different dimensions of all harms observed in trials. Except for the tendril plot, all other plots do not require individual participant data. The dot plot and volcano plot are favored as visualization approaches to present an overall summary of harms data. Our value assessment found the dot plot and volcano plot were favored by content experts. Using visualizations to report harms could improve communication. Trialists can use our provided code to easily implement these approaches.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Designing clinically useful psychopharmacological trials: challenges and ways forward

The clinical guidelines that underpin the use of drugs for mental disorders are informed by evidence from randomised controlled trials (RCTs). RCTs are performed to obtain marketing authorisation from regulators. The methods used in these RCTs could be appropriate for early phases of drug development because they identify drugs with important harms and drugs that are efficacious for specific health problems and populations. RCTs done before marketing authorisation do not tend to address clinical questions that concern the effectiveness of a drug in heterogeneous and comorbid populations, the optimisation of drug sequencing and discontinuation, or the comparative benefits and harms of different drugs that could be used for the same health problem. This Review proposes an overview of some shortcomings of RCTs, at an individual level and at the whole portfolio level, and identifies some methods in planning, conducting, and carrying out analyses in RCTs that could enhance their ability to support therapeutic decisions. These suggestions include: identifying patient-important questions to be investigated by psychopharmacological RCTs; embedding pragmatic RCTs within clinical practice to improve generalisability to target populations; collecting evidence about drugs in overlooked populations; developing methods to facilitate the recruitment of patients with mental disorders and to reduce the number of patients who drop out, using specific methods; using core outcome sets to standardise the assessment of benefits and harms; and recording systematically serious objective outcomes, such as suicide or hospitalisation, to be evaluated in meta-analyses. This work is a call to address questions relevant to patients using diverse design of RCTs, thus contributing to the development of a patient-centred, evidence-based psychiatry.

Vitamin A supplementation for preventing morbidity and mortality in children from six months to five years of age

BACKGROUND

Vitamin A deficiency (VAD) is a major public health problem in low- and middle-income countries, affecting 190 million children under five years of age and leading to many adverse health consequences, including death. Based on prior evidence and a previous version of this review, the World Health Organization has continued to recommend vitamin A supplementation (VAS) for children aged 6 to 59 months. The last version of this review was published in 2017, and this is an updated version of that review.

OBJECTIVES

To assess the effects of vitamin A supplementation (VAS) for preventing morbidity and mortality in children aged six months to five years.

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, six other databases, and two trials registers up to March 2021. We also checked reference lists and contacted relevant organisations and researchers to identify additional studies.

SELECTION CRITERIA

Randomised controlled trials (RCTs) and cluster-RCTs evaluating the effect of synthetic VAS in children aged six months to five years living in the community. We excluded studies involving children in hospital and children with disease or infection. We also excluded studies evaluating the effects of food fortification, consumption of vitamin A rich foods, or beta-carotene supplementation.

DATA COLLECTION AND ANALYSIS

For this update, two review authors independently assessed studies for inclusion resolving discrepancies by discussion. We performed meta-analyses for outcomes, including all-cause and cause-specific mortality, disease, vision, and side effects. We used the GRADE approach to assess the quality of the evidence.

MAIN RESULTS

The updated search identified no new RCTs. We identified 47 studies, involving approximately 1,223,856 children. Studies were set in 19 countries: 30 (63%) in Asia, 16 of these in India; 8 (17%) in Africa; 7 (15%) in Latin America, and 2 (4%) in Australia. About one-third of the studies were in urban/periurban settings, and half were in rural settings; the remaining studies did not clearly report settings. Most studies included equal numbers of girls and boys and lasted about one year. The mean age of the children was about 33 months. The included studies were at variable overall risk of bias; however, evidence for the primary outcome was at low risk of bias. A meta-analysis for all-cause mortality included 19 trials (1,202,382 children). At longest follow-up, there was a 12% observed reduction in the risk of all-cause mortality for VAS compared with control using a fixed-effect model (risk ratio (RR) 0.88, 95% confidence interval (CI) 0.83 to 0.93; high-certainty evidence). Nine trials reported mortality due to diarrhoea and showed a 12% overall reduction for VAS (RR 0.88, 95% CI 0.79 to 0.98; 1,098,538 children; high-certainty evidence). There was no evidence of a difference for VAS on mortality due to measles (RR 0.88, 95% CI 0.69 to 1.11; 6 studies, 1,088,261 children; low-certainty evidence), respiratory disease (RR 0.98, 95% CI 0.86 to 1.12; 9 studies, 1,098,538 children; low-certainty evidence), and meningitis. VAS reduced the incidence of diarrhoea (RR 0.85, 95% CI 0.82 to 0.87; 15 studies, 77,946 children; low-certainty evidence), measles (RR 0.50, 95% CI 0.37 to 0.67; 6 studies, 19,566 children; moderate-certainty evidence), Bitot's spots (RR 0.42, 95% CI 0.33 to 0.53; 5 studies, 1,063,278 children; moderate-certainty evidence), night blindness (RR 0.32, 95% CI 0.21 to 0.50; 2 studies, 22,972 children; moderate-certainty evidence), and VAD (RR 0.71, 95% CI 0.65 to 0.78; 4 studies, 2262 children, moderate-certainty evidence). However, there was no evidence of a difference on incidence of respiratory disease (RR 0.99, 95% CI 0.92 to 1.06; 11 studies, 27,540 children; low-certainty evidence) or hospitalisations due to diarrhoea or pneumonia. There was an increased risk of vomiting within the first 48 hours of VAS (RR 1.97, 95% CI 1.44 to 2.69; 4 studies, 10,541 children; moderate-certainty evidence).

AUTHORS' CONCLUSIONS

This update identified no new eligible studies and the conclusions remain the same. VAS is associated with a clinically meaningful reduction in morbidity and mortality in children. Further placebo-controlled trials of VAS in children between six months and five years of age would not change the conclusions of this review, although studies that compare different doses and delivery mechanisms are needed. In populations with documented VAD, it would be unethical to conduct placebo-controlled trials.

Harms in Systematic Reviews Paper 3: Given the same data sources, systematic reviews of gabapentin have different results for harms

OBJECTIVE

In this methodologic study (Part 2 of 2), we examined the overlap in sources of evidence and the corresponding results for harms in systematic reviews for gabapentin.

STUDY DESIGN & SETTING

We extracted all citations referenced as sources of evidence for harms of gabapentin from 70 systematic reviews, as well as the harms assessed and numerical results. We assessed consistency of harms between pairs of reviews with a high degree of overlap in sources of evidence (>50%) as determined by corrected covered area (CCA).

RESULTS

We found 514 reports cited across 70 included reviews. Most reports (244/514, 48%) were not cited in more than one review. Among 18 pairs of reviews, we found reviews had differences in which harms were assessed and their choice to meta-analyze estimates or present descriptive summaries. When a specific harm was meta-analyzed in a pair of reviews, we found similar effect estimates.

CONCLUSION

Differences in harms results across reviews can occur because the choice of harms is driven by reviewer preferences, rather than standardized approaches to selecting harms for assessment. A paradigm shift is needed in the current approach to synthesizing harms.

Harms in Systematic Reviews Paper 2: Methods used to assess harms are neglected in systematic reviews of gabapentin

OBJECTIVE

We compared methods used with current recommendations for synthesizing harms in systematic reviews and meta-analyses (SRMAs) of gabapentin.

STUDY DESIGN & SETTING

We followed recommended systematic review practices. We selected reliable SRMAs of gabapentin (i.e., met a pre-defined list of methodological criteria) that assessed at least one harm. We extracted and compared methods in four areas: pre-specification, searching, analysis, and reporting. Whereas our focus in this paper is on the methods used, Part 2 examines the results for harms across reviews.

RESULTS

We screened 4320 records and identified 157 SRMAs of gabapentin, 70 of which were reliable. Most reliable reviews (51/70; 73%) reported following a general guideline for SRMA conduct or reporting, but none reported following recommendations specifically for synthesizing harms. Across all domains assessed, review methods were designed to address questions of benefit and rarely included the additional methods that are recommended for evaluating harms.

CONCLUSION

Approaches to assessing harms in SRMAs we examined are tokenistic and unlikely to produce valid summaries of harms to guide decisions. A paradigm shift is needed. At a minimal, reviewers should describe any limitations to their assessment of harms and provide clearer descriptions of methods for synthesizing harms.

Harms in Systematic Reviews Paper 1: An introduction to research on harms

OBJECTIVE

Most systematic reviews of interventions focus on potential benefits. Common methods and assumptions that are appropriate for assessing benefits can be inappropriate for harms. This paper provides a primer on researching harms, particularly in systematic reviews.

STUDY DESIGN AND SETTING

Commentary describing challenges with assessing harm.

RESULTS

Investigators should be familiar with various terminologies used to describe, classify, and group harms. Published reports of clinical trials include limited information about harms, so systematic reviewers should not depend on these studies and journal articles to reach conclusions about harms. Visualizations might improve communication of multiple dimensions of harms such as severity, relatedness, and timing.

CONCLUSION

The terminology, classification, detection, collection, and reporting of harms create unique challenges that take time, expertise, and resources to navigate in both primary studies and evidence syntheses. Systematic reviewers might reach incorrect conclusions if they focus on evidence about harms found in published reports of randomized trials of a particular health problem. Systematic reviews could be improved through better identification and reporting of harms in primary studies and through better training and uptake of appropriate methods for synthesizing evidence about harms.

Transparent, Open, and Reproducible Prevention Science

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

Modern Meta-Analytic Methods in Prevention Science: Introduction to the Special Issue

Meta-analyses that statistically synthesize evidence from multiple research studies can play an important role in advancing evidence-informed prevention science. When done in the context of a well-conducted systematic review, meta-analysis is a powerful tool for synthesizing evidence and exploring research questions that are difficult to address in individual studies, such as the association of individual study limitations on intervention effect estimates, replicability of empirical findings, and variation of effect estimates across populations and settings. Alongside the rapid growth in the number of published reviews and meta-analyses, there has been a parallel growth in the development of meta-analytic techniques to handle the increasingly complex types of questions and types of evidence relevant to prevention science. Despite this rapid evolution of meta-analytic techniques and approaches, there is still a lag between the development of new techniques and their uptake by researchers in the field. This paper serves as a brief introduction to this special issue of Prevention Science, entitled "Modern Meta-Analytic Methods in Prevention Science," which highlights recent developments in meta-analytic methods and demonstrates their application to prevention research. This special issue makes an important contribution to the field by ensuring these methodological advances are widely accessible to prevention science researchers, thereby improving their uptake and utilization, and ultimately improving the utility and rigor of research syntheses for informing evidence-based decision making in prevention.

Peer review reduces spin in PCORI research reports

Background

The Patient-Centered Outcomes Research Institute (PCORI) is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. PCORI peer review emphasizes adherence to PCORI’s Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin.

Methods

Two independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion of reports in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers’ comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles.

Results

We included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 (86%) submitted research reports. Types of spin included reporting bias (46/55; 84%), inappropriate interpretation (40/55; 73%), inappropriate extrapolation of results (15/55; 27%), and inappropriate attribution of causality (5/55; 9%). Authors addressed comments about spin related to 47/55 (85%) of the reports.

Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 (40%) articles, of which 27/44 (61%) contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review (63% vs 58%).

Discussion

Just as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder’s peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41073-021-00119-1.

The PRISMA 2020 statement: an updated guideline for reporting systematic reviews

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews. Full English text available from:www.revespcardiol.org/en.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Restoring invisible and abandoned trials of gabapentin for neuropathic pain: a clinical and methodological investigation

INTRODUCTION

Gabapentin (Neurontin) is prescribed widely for conditions for which it has not been approved by regulators, including certain neuropathic pain conditions. There is limited evidence that gabapentin is safe and effective for the treatment of neuropathic pain. Published trial reports, and systematic reviews based on published trial reports, mislead patients and providers because information about gabapentin's harms has been published only partly. We confirmed that trials conducted by the drug developer have been abandoned, and we plan to conduct a restoration with support from the Restoring Invisible and Abandoned Trials Support Centre (https://restoringtrials.org/).

METHODS AND ANALYSIS

In this study, we will analyse and report the harms that were observed in six trials of gabapentin, which have not been reported publicly (eg, in journal articles). We will use clinical study reports and individual participant data to identify and report the harms observed in each individual trial and to summarise the harms observed across all six trials. We will report all adverse events observed in the included trials by sharing deidentified data and summary tables on the Open Science Framework (https://osf.io/w8puv/). Additionally, we will produce a summary report that describes differences between the randomised groups in each trial and across trials for prespecified harms outcomes.

ETHICS AND DISSEMINATION

We will use secondary data. This study was determined to be exempt from Institutional Review Board (IRB) review (protocol #1910607198).

Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol

Background : Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists. Objective : To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions. Methods : We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals. Conclusion : The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.

Evaluating implementation of the Transparency and Openness Promotion (TOP) guidelines: the TRUST process for rating journal policies, procedures, and practices

BACKGROUND

The Transparency and Openness Promotion (TOP) Guidelines describe modular standards that journals can adopt to promote open science. The TOP Factor is a metric to describe the extent to which journals have adopted the TOP Guidelines in their policies. Systematic methods and rating instruments are needed to calculate the TOP Factor. Moreover, implementation of these open science policies depends on journal procedures and practices, for which TOP provides no standards or rating instruments.

METHODS

We describe a process for assessing journal policies, procedures, and practices according to the TOP Guidelines. We developed this process as part of the Transparency of Research Underpinning Social Intervention Tiers (TRUST) Initiative to advance open science in the social intervention research ecosystem. We also provide new instruments for rating journal instructions to authors (policies), manuscript submission systems (procedures), and published articles (practices) according to standards in the TOP Guidelines. In addition, we describe how to determine the TOP Factor score for a journal, calculate reliability of journal ratings, and assess coherence among a journal's policies, procedures, and practices. As a demonstration of this process, we describe a protocol for studying approximately 345 influential journals that have published research used to inform evidence-based policy.

DISCUSSION

The TRUST Process includes systematic methods and rating instruments for assessing and facilitating implementation of the TOP Guidelines by journals across disciplines. Our study of journals publishing influential social intervention research will provide a comprehensive account of whether these journals have policies, procedures, and practices that are consistent with standards for open science and thereby facilitate the publication of trustworthy findings to inform evidence-based policy. Through this demonstration, we expect to identify ways to refine the TOP Guidelines and the TOP Factor. Refinements could include: improving templates for adoption in journal instructions to authors, manuscript submission systems, and published articles; revising explanatory guidance intended to enhance the use, understanding, and dissemination of the TOP Guidelines; and clarifying the distinctions among different levels of implementation. Research materials are available on the Open Science Framework: https://osf.io/txyr3/ .

Creating a Program to Support Registering and Reporting Clinical Trials at Johns Hopkins University

PROBLEM

The Food and Drug Administration Amendments Act of 2007 (FDAAA) and the National Institutes of Health (NIH) require that many clinical trials register and report results on ClinicalTrials.gov. Noncompliance with these policies denies research participants and scientists access to potentially relevant findings and could lead to monetary penalties or loss of funding. After discovering hundreds of potentially noncompliant trials affiliated with the institution, the Johns Hopkins University School of Medicine (JHUSOM) sought to develop a program to support research teams with registration and reporting requirements.

APPROACH

JHUSOM conducted a baseline assessment of institutional compliance in 2015, launched the ClinicalTrials.gov Program in June 2016, and expanded the program to the Sidney Kimmel Comprehensive Cancer Center in April 2018. The program is innovative in its comprehensive approach, and it was among the first to bring a large number of trials into compliance.

OUTCOMES

From September 2015 to September 2020, JHUSOM brought completed and ongoing trials into compliance with FDAAA and NIH policies and maintained almost perfect compliance for new trials. During this period, the proportion of trials potentially noncompliant with the FDAAA decreased from 44% (339/774) to 2% (32/1,304).

NEXT STEPS

JHUSOM continues to develop and evaluate tools and procedures that facilitate trial registration and results reporting. In collaboration with other academic medical centers, JHUSOM plans to share resources and to identify and disseminate best practices. This report identifies practical lessons for institutions that might develop similar programs.

The PRISMA 2020 statement: An updated guideline for reporting systematic reviews

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.

PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews

The methods and results of systematic reviews should be reported in sufficient detail to allow users to assess the trustworthiness and applicability of the review findings. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement was developed to facilitate transparent and complete reporting of systematic reviews and has been updated (to PRISMA 2020) to reflect recent advances in systematic review methodology and terminology. Here, we present the explanation and elaboration paper for PRISMA 2020, where we explain why reporting of each item is recommended, present bullet points that detail the reporting recommendations, and present examples from published reviews. We hope that changes to the content and structure of PRISMA 2020 will facilitate uptake of the guideline and lead to more transparent, complete, and accurate reporting of systematic reviews.