Getting Over TOP

Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals

OBJECTIVE

The goals of this study were to evaluate the extent that physical therapist journals support open science research practices by adhering to the Transparency and Openness Promotion (TOP) guidelines and to assess the relationship between journal scores and their respective journal impact factor (JIF).

METHODS

Scimago, mapping studies, the National Library of Medicine, and journal author guidelines were searched to identify physical therapist journals for inclusion. Journals were graded on 10 standards (29 available total points) related to transparency with data, code, research materials, study design and analysis, preregistration of studies and statistical analyses, replication, and open science badges. The relationship between journal transparency and openness scores and their JIF was determined.

RESULTS

Thirty-five journals' author guidelines were assigned transparency and openness factor scores. The median score (interquartile range) across journals was 3.00 out of 29 (3.00) points (for all journals the scores ranged from 0 to 8). The 2 standards with the highest degree of implementation were design and analysis transparency (reporting guidelines) and study preregistration. No journals reported on code transparency, materials transparency, replication, and open science badges. TOP factor scores were a significant predictor of JIF scores.

CONCLUSION

There is low implementation of the TOP standards by physical therapist journals. TOP factor scores demonstrated predictive abilities for JIF scores. Policies from journals must improve to make open science practices the standard in research. Journals are in an influential position to guide practices that can improve the rigor of publication which, ultimately, enhances the evidence-based information used by physical therapists.

IMPACT

Transparent, open, and reproducible research will move the profession forward by improving the quality of research and increasing the confidence in results for implementation in clinical care.

Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors

The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.

Considerations towards the better integration of epidemiology into quantitative risk assessment

Environmental epidemiology has proven critical to study various associations between environmental exposures and adverse human health effects. However, there is a perception that it often does not sufficiently inform quantitative risk assessment. To help address this concern, in 2017, the Health and Environmental Sciences Institute initiated a project engaging the epidemiology, exposure science, and risk assessment communities with tripartite representation from government agencies, industry, and academia, in a dialogue on the use of environmental epidemiology for quantitative risk assessment and public health decision making. As part of this project, four meetings attended by experts in epidemiology, exposure science, toxicology, statistics, and risk assessment, as well as one additional meeting engaging funding agencies, were organized to explore incentives and barriers to realizing the full potential of epidemiological data in quantitative risk assessment. A set of questions was shared with workshop participants prior to the meetings, and two case studies were used to support the discussion. Five key ideas emerged from these meetings as areas of desired improvement to ensure that human data can more consistently become an integral part of quantitative risk assessment: 1) reducing confirmation and publication bias, 2) increasing communication with funding agencies to raise awareness of research needs, 3) developing alternative funding channels targeted to support quantitative risk assessment, 4) making data available for reuse and analysis, and 5) developing cross-disciplinary and cross-sectoral interactions, collaborations, and training. We explored and integrated these themes into a roadmap illustrating the need for a multi-stakeholder effort to ensure that epidemiological data can fully contribute to the quantitative evaluation of human health risks, and to build confidence in a reliable decision-making process that leverages the totality of scientific evidence.

Open science and conflict of interest policies of medical and health sciences journals before and during the COVID-19 pandemic: A repeat cross-sectional study: Open science policies of medical journals

Objectives

To audit the transparent and open science standards of health and medical sciences journal policies and explore the impact of the COVID-19 pandemic.

Design

Repeat cross-sectional study.

Setting

19 journals listed in Google Scholar's Top Publications for health and medical sciences.

Participants

Blood, Cell, Circulation, European Heart Journal, Gastroenterology, Journal of Clinical Oncology, Journal of the American College of Cardiology, Nature Genetics, Nature Medicine, Nature Neuroscience, Neuron, PLoS ONE, Proceedings of the National Academy of Sciences, Science Translational Medicine, The British Medical Journal, The Journal of the American Medical Association, The Lancet, The Lancet Oncology, and The New England Journal of Medicine.

Main outcome measures

We used the Transparency and Openness Promotion (TOP) guideline and the International Committee of Medical Journal Editors (ICMJE) requirements for disclosing conflicts of interest (COIs) to evaluate journals standards.

Results

TOP scores slightly improved during the COVID-19 pandemic, from a median of 5 (IQR: 2-12.5) out of a possible 24 points in February 2020 to 7 (IQR: 4-12) in May 2021, but overall, scores were very low at both time points. Journal policies scored highest for their adherence to data transparency and scored lowest for preregistration of study protocols and analysis plans and the submission of replication studies. Most journals fulfilled all ICMJE provisions for reporting COIs before (84%; n  =  16) and during (95%; n  =  18) the COVID-19 pandemic.

Conclusions

The COVID-19 pandemic has highlighted the importance of practising open science. However, requirements for open science practices in audited policies were overall low, which may impede progress in health and medical research. As key stakeholders in disseminating research, journals should promote a research culture of greater transparency and more robust open science practices.

Feasibility of an Audit and Feedback Intervention to Facilitate Journal Policy Change Towards Greater Promotion of Transparency and Openness in Sports Science Research

OBJECTIVES

To evaluate (1) the feasibility of an audit-feedback intervention to facilitate sports science journal policy change, (2) the reliability of the Transparency of Research Underpinning Social Intervention Tiers (TRUST) policy evaluation form, and (3) the extent to which policies of sports science journals support transparent and open research practices.

METHODS

We conducted a cross-sectional, audit-feedback, feasibility study of transparency and openness standards of the top 38 sports science journals by impact factor. The TRUST form was used to evaluate journal policies support for transparent and open research practices. Feedback was provided to journal editors in the format of a tailored letter. Inter-rater reliability and agreement of the TRUST form was assessed using intraclass correlation coefficients and the standard error of measurement, respectively. Time-based criteria, fidelity of intervention delivery and qualitative feedback were used to determine feasibility.

RESULTS

The audit-feedback intervention was feasible based on the time taken to rate journals and provide tailored feedback. The mean (SD) score on the TRUST form (range 0-27) was 2.05 (1.99), reflecting low engagement with transparent and open practices. Inter-rater reliability of the overall score of the TRUST form was moderate [ICC (2,1) = 0.68 (95% CI 0.55-0.79)], with standard error of measurement of 1.17. However, some individual items had poor reliability.

CONCLUSION

Policies of the top 38 sports science journals have potential for improved support for transparent and open research practices. The feasible audit-feedback intervention developed here warrants large-scale evaluation as a means to facilitate change in journal policies.

REGISTRATION

OSF ( https://osf.io/d2t4s/ ).