Transparent, Open, and Reproducible Prevention Science

A scoping review on what constitutes a good research culture

Background

The crisis in research culture is well documented, covering issues such as a tendency for quantity over quality, unhealthy competitive environments, and assessment based on publications, journal prestige and funding. In response, research institutions need to assess their own practices to promote and advocate for change in the current research ecosystem. The purpose of the scoping review was to explore ' What does the evidence say about the 'problem' with 'poor' research culture, what are the benefits of 'good' research culture, and what does 'good' look like?'

Aims

To examine the peer-reviewed and grey literature to explore the interplay between research culture, open research, career paths, recognition and rewards, and equality, diversity, and inclusion, as part of a larger programme of activity for a research institution.

Methods

A scoping review was undertaken. Six databases were searched along with grey literature. Eligible literature had relevance to academic research institutions, addressed research culture, and were published between January 2017 to May 2022. Evidence was mapped and themed to specific categories. The search strategy, screening and analysis took place between April-May 2022.

Results

1666 titles and abstracts, and 924 full text articles were assessed for eligibility. Of these, 253 articles met the eligibility criteria for inclusion. A purposive sampling of relevant websites was drawn from to complement the review, resulting in 102 records included in the review. Key areas for consideration were identified across the four themes of job security, wellbeing and equality of opportunity, teamwork and interdisciplinary, and research quality and accountability.

Conclusions

There are opportunities for research institutions to improve their own practice, however institutional solutions cannot act in isolation. Research institutions and research funders need to work together to build a more sustainable and inclusive research culture that is diverse in nature and supports individuals' well-being, career progression and performance.

A pathway to strengthening open science: comments on the draft South African Ethics in Health Research Guidelines

The recently released draft South African Ethics in Health Research Guidelines: Principles, Processes and Structures (Draft Guidelines) by the National Health Research Ethics Council recognize open data and provide guiding principles for this in the context of health research in South Africa. While its inclusion is a positive development, there is room for improvement. Although the Draft Guidelines leverage the Draft National Policy on Data and Cloud, it lacks incorporation of other relevant government policies, notably the Draft National Open Science Policy, and fails to sufficiently detail the principles of open science and open access. This limited scope and lack of comprehensive definition and detailed guidance present challenges for researchers in conducting ethical and responsible health research in South Africa. It constrains the Draft Guidelines from fully aligning with national imperatives and from fostering African-centric approaches. To address these issues, it is recommended that the Draft Guidelines integrate broader policies and principles, enhance clarity through comprehensive definitions, provide detailed guidance on open access, and promote African-centric approaches. Implementing these solutions will strengthen the Draft Guidelines, aligning them with national visions of open science, and thereby harnessing the full potential of South Africa's diverse scientific community in advancing health research.

Advocating for Generalizability: Accepting Inherent Variability in Translation of Animal Research Outcomes

Advancing scientific discovery requires investigators to embrace research practices that increase transparency and disclosure about materials, methods, and outcomes. Several research advocacy and funding organizations have produced guidelines and recommended practices to enhance reproducibility through detailed and rigorous research approaches; however, confusion around vocabulary terms and a lack of adoption of suggested practices have stymied successful implementation. Although reproducibility of research findings cannot be guaranteed due to extensive inherent variables in attempts at experimental repetition, the scientific community can advocate for generalizability in the application of data outcomes to ensure a broad and effective impact on the comparison of animals to translation within human research. This report reviews suggestions, based upon work with National Institutes of Health advisory groups, for improving rigor and transparency in animal research through aspects of experimental design, statistical assessment, and reporting factors to advocate for generalizability in the application of comparative outcomes between animals and humans.

Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals

OBJECTIVE

The goals of this study were to evaluate the extent that physical therapist journals support open science research practices by adhering to the Transparency and Openness Promotion (TOP) guidelines and to assess the relationship between journal scores and their respective journal impact factor (JIF).

METHODS

Scimago, mapping studies, the National Library of Medicine, and journal author guidelines were searched to identify physical therapist journals for inclusion. Journals were graded on 10 standards (29 available total points) related to transparency with data, code, research materials, study design and analysis, preregistration of studies and statistical analyses, replication, and open science badges. The relationship between journal transparency and openness scores and their JIF was determined.

RESULTS

Thirty-five journals' author guidelines were assigned transparency and openness factor scores. The median score (interquartile range) across journals was 3.00 out of 29 (3.00) points (for all journals the scores ranged from 0 to 8). The 2 standards with the highest degree of implementation were design and analysis transparency (reporting guidelines) and study preregistration. No journals reported on code transparency, materials transparency, replication, and open science badges. TOP factor scores were a significant predictor of JIF scores.

CONCLUSION

There is low implementation of the TOP standards by physical therapist journals. TOP factor scores demonstrated predictive abilities for JIF scores. Policies from journals must improve to make open science practices the standard in research. Journals are in an influential position to guide practices that can improve the rigor of publication which, ultimately, enhances the evidence-based information used by physical therapists.

IMPACT

Transparent, open, and reproducible research will move the profession forward by improving the quality of research and increasing the confidence in results for implementation in clinical care.

Introduction to the Special Issue on Innovations and Applications of Integrative Data Analysis (IDA) and Related Data Harmonization Procedures in Prevention Science

This paper serves as an introduction to the special issue of Prevention Science entitled, "Innovations and Applications of Integrative Data Analysis (IDA) and Related Data Harmonization Procedures in Prevention Science." This special issue includes a collection of original papers from multiple disciplines that apply individual-level data synthesis methodologies, including IDA, individual participant meta-analysis, and other related methods to harmonize and integrate multiple datasets from intervention trials of the same or similar interventions. This work builds on a series of papers appearing in a prior Prevention Science special issue, entitled "Who Benefits from Programs to Prevent Adolescent Depression?" (Howe, Pantin, & Perrino, 2018). Since the publication of this prior work, the use of individual-level data synthesis has increased considerably in and outside of prevention. As such, there is a need for an update on current and future directions in IDA, with careful consideration of innovations and applications of these methods to fill important research gaps in prevention science. The papers in this issue are organized into two broad categories of (1) evidence synthesis papers that apply best practices in data harmonization and individual-level data synthesis and (2) new and emerging design, psychometric, and methodological issues and solutions. This collection of original papers is followed by two invited commentaries which provide insight and important reflections on the field and future directions for prevention science.

Open Science Standards at Journals that Inform Evidence-Based Policy

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

The use and acceptability of preprints in health and social care settings: A scoping review

BACKGROUND

Preprints are open and accessible scientific manuscript or report that is shared publicly, through a preprint server, before being submitted to a journal. The value and importance of preprints has grown since its contribution during the public health emergency of the COVID-19 pandemic. Funders and publishers are establishing their position on the use of preprints, in grant applications and publishing models. However, the evidence supporting the use and acceptability of preprints varies across funders, publishers, and researchers. The scoping review explored the current evidence on the use and acceptability of preprints in health and social care settings by publishers, funders, and the research community throughout the research lifecycle.

METHODS

A scoping review was undertaken with no study or language limits. The search strategy was limited to the last five years (2017-2022) to capture changes influenced by COVID-19 (e.g., accelerated use and role of preprints in research). The review included international literature, including grey literature, and two databases were searched: Scopus and Web of Science (24 August 2022).

RESULTS

379 titles and abstracts and 193 full text articles were assessed for eligibility. Ninety-eight articles met eligibility criteria and were included for full extraction. For barriers and challenges, 26 statements were grouped under four main themes (e.g., volume/growth of publications, quality assurance/trustworthiness, risks associated to credibility, and validation). For benefits and value, 34 statements were grouped under six themes (e.g., openness/transparency, increased visibility/credibility, open review process, open research, democratic process/systems, increased productivity/opportunities).

CONCLUSIONS

Preprints provide opportunities for rapid dissemination but there is a need for clear policies and guidance from journals, publishers, and funders. Cautionary measures are needed to maintain the quality and value of preprints, paying particular attention to how findings are translated to the public. More research is needed to address some of the uncertainties addressed in this review.

Neurocognitive and self-reported psychosocial and behavioral functioning in siblings of individuals with neurodevelopmental conditions: a study using remote self-administered testing

OBJECTIVE

This study compared and explored the neurocognitive profiles of siblings of persons with and without neurodevelopmental conditions (NDCs) and associations between objective test performance and self-reported psychosocial functioning.

METHODS

Siblings of persons with and without NDCs (64 NDC and 64 control siblings; mean age 19.88 years, range 11-27 years, 73.44% female, 75.78% White Caucasian) completed self-report questionnaires and self-administered computerized neurocognitive tests of executive functioning (EF). Using Bayesian analyses, we examined cross-sectional associations between self-reported psychosocial functioning and cognitive test performance, and predictors of EF over 15 months.

RESULTS

NDC siblings had poorer working memory, inhibition, attention, and shifting compared to controls, as measured by experimental paradigms on the backward Corsi span, N-Back 2-back task, Stop Signal Task, Sustained Attention to Response Task, and the Wisconsin Card Sorting Test (effect size δ ranging 0.49 to 0.64). Bayesian cross-sectional networks revealed negative emotion reactivity and working memory difficulties were central to the NDC sibling network. Over 15 months, poorer EF (k low test scores) was predicted by negative emotion reactivity, sleep problems, and anxiety, over and above effects of age and subclinical autistic and ADHD traits. Siblings of autistic individuals and persons with fetal alcohol spectrum disorder had higher rates of neurocognitive and psychiatric difficulties than other NDCs and controls (Bayes factors >20).

CONCLUSIONS

Neurocognitive difficulties were associated with transdiagnostic vulnerability to poorer wellbeing in NDC siblings. These findings demonstrate the feasibility of remote online cognitive testing and highlight the importance of individualized prevention and intervention for NDC siblings.

Psychosocial and neurocognitive correlates of suicidal thoughts and behaviours amongst siblings of persons with and without neurodevelopmental conditions

OBJECTIVE

Siblings of individuals with neurodevelopmental conditions (NDCs) have greater incidence of neuropsychiatric diagnoses and neurocognitive difficulties compared to siblings of persons without NDCs. Despite suicidality being labelled a global health crisis (WHO, 2014) and NDC siblings experiencing risk factors implicated in suicidality, no previous studies examined suicidality amongst adolescent and young adult siblings of persons with NDCs. Our study aimed to bridge this gap.

METHOD

The present study used Bayesian analyses and risk classification models to examine individual and environmental risk factors associated with suicidal thoughts and behaviours amongst siblings of persons with and without NDCs (n = 267; 132 NDC, 135 control group, mean age 20.61, range 14-27, 76.40% female, 76.78% White Caucasian), as measured using self-report survey data and remote self-administered cognitive tests.

RESULTS

NDC siblings had higher rates of current nonsuicidal self-injury (NSSI; 18.94% versus 14.07%, δ = -0.32), suicidal ideation (25.76% versus 8.89%, δ = -0.40) and history of suicide attempts (18.18% versus 4.44%, δ = -0.43) compared to controls. Classification models using boosting and random forest demonstrated adequate performance: positive predictive value 0.86-0.91, negative predictive value 0.81-0.90, false negative rates 0.11-0.24. Cognitive inflexibility, alexithymia, inattention, bullying, depression, NSSI, and eating or psychotic disorder history had the highest relative importance in predicting lifetime suicidality. Poorer executive functioning (measured by the Wisconsin Card Sorting Task, Sustained Attention to Response Task, Stop Signal Task, and N-Back 2-back task) was strongly correlated with suicidality.

CONCLUSIONS

Screening for proximal and modifiable risk factors is critical to inform suicidal behaviour intervention and prevention programs for at-risk siblings.

Iranian researchers' perspective about concept and effect of open science on research publication

BACKGROUND

Sharing research outputs with open science methods for different stakeholders causes better access to different studies to solve problems in diverse fields, which leads to equal access conditions to research resources, as well as greater scientific productivity. Therefore, the aim of this study was to perceive the concept of openness in research among Iranian health researchers.

METHODS

From the beginning of August to the middle of November 2021, twenty semi-structured interviews were held with Iranian health researchers from different fields using purposeful, snowball, and convenience sampling. The interviews continued until data saturation. Data analysis was performed with thematic analysis using MAXQDA 20. Finally, seven main issues related to open science were identified.

RESULTS

Through analysis of the interviews, 235 primary codes and 173 main codes were extracted in 22 subclasses. After careful evaluation and integration of subclasses and classes, they were finally classified into nine categories and three main themes. Analysis showed that openness in research was related to three main themes: researchers' understanding of open science, the impact of open science on publication and sharing of research, concerns and reluctance to open research.

CONCLUSION

The conditions of access to research output should be specified given the diversity of studies conducted in the field of health; issues like privacy as an important topic of access to data and information in the health system should also be specified. Our analysis indicated that the conditions of publication and sharing of research processes should be stated according to different scopes of health fields. The concept of open science was related to access to findings and other research items regardless of cost, political, social, or racial barriers, which could create collective wisdom in the development of knowledge. The process of publication and sharing of research related to open access applies to all types of outputs, conditions of access, increasing trust in research, creation of diverse publication paths, and broader participation of citizens in research. Open science practices should be promoted to increase the circulation and exploitation rates of knowledge while adjusting and respecting the limits of privacy, intellectual property and national security rights of countries.

Reproducibility and Scientific Integrity of Big Data Research in Urban Public Health and Digital Epidemiology: A Call to Action

The emergence of big data science presents a unique opportunity to improve public-health research practices. Because working with big data is inherently complex, big data research must be clear and transparent to avoid reproducibility issues and positively impact population health. Timely implementation of solution-focused approaches is critical as new data sources and methods take root in public-health research, including urban public health and digital epidemiology. This commentary highlights methodological and analytic approaches that can reduce research waste and improve the reproducibility and replicability of big data research in public health. The recommendations described in this commentary, including a focus on practices, publication norms, and education, are neither exhaustive nor unique to big data, but, nonetheless, implementing them can broadly improve public-health research. Clearly defined and openly shared guidelines will not only improve the quality of current research practices but also initiate change at multiple levels: the individual level, the institutional level, and the international level.

Adoption of Transparency and Openness Promotion (TOP) Guidelines across Journals

Journal policies continuously evolve to enable knowledge sharing and support reproducible science. However, that change happens within a certain framework. Eight modular standards with three levels of increasing stringency make Transparency and Openness Promotion (TOP) guidelines which can be used to evaluate to what extent and with which stringency journals promote open science. Guidelines define standards for data citation, transparency of data, material, code and design and analysis, replication, plan and study pre-registration, and two effective interventions: “Registered reports” and “Open science badges”, and levels of adoption summed up across standards define journal’s TOP Factor. In this paper, we analysed the status of adoption of TOP guidelines across two thousand journals reported in the TOP Factor metrics. We show that the majority of the journals’ policies align with at least one of the TOP’s standards, most likely “Data citation” (70%) followed by “Data transparency” (19%). Two-thirds of adoptions of TOP standard are of the stringency Level 1 (less stringent), whereas only 9% is of the stringency Level 3. Adoption of TOP standards differs across science disciplines and multidisciplinary journals (N = 1505) and journals from social sciences (N = 1077) show the greatest number of adoptions. Improvement of the measures that journals take to implement open science practices could be done: (1) discipline-specific, (2) journals that have not yet adopted TOP guidelines could do so, (3) the stringency of adoptions could be increased.