Use of reproducible research practices in public health: A survey of public health analysts

A guide for social science journal editors on easing into open science

Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org ) has collated several resources on embedding open science in journal editing ( www.dpjedi.org/resources ). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx ).

German funders' data sharing policies-A qualitative interview study

BACKGROUND

Data sharing is commonly seen as beneficial for science but is not yet common practice. Research funding agencies are known to play a key role in promoting data sharing, but German funders' data sharing policies appear to lag behind in international comparison. This study aims to answer the question of how German data sharing experts inside and outside funding agencies perceive and evaluate German funders' data sharing policies and overall efforts to promote data sharing.

METHODS

This study is based on sixteen guided expert interviews with representatives of German funders and German research data experts from stakeholder organisations, who shared their perceptions of German' funders efforts to promote data sharing. By applying the method of qualitative content analysis to our interview data, we categorise and describe noteworthy aspects of the German data sharing policy landscape and illustrate our findings with interview passages.

RESULTS

We present our findings in five sections to distinguish our interviewees' perceptions on a) the status quo of German funders' data sharing policies, b) the role of funders in promoting data sharing, c) current and potential measures by funders to promote data sharing, d) general barriers to those measures, and e) the implementation of more binding data sharing requirements.

DISCUSSION AND CONCLUSION

Although funders are perceived to be important promoters and facilitators of data sharing throughout our interviews, only few German funding agencies have data sharing policies in place. Several interviewees stated that funders could do more, for example by providing incentives for data sharing or by introducing more concrete policies. Our interviews suggest the academic freedom of grantees is widely perceived as an obstacle for German funders in introducing mandatory data sharing requirements. However, some interviewees stated that stricter data sharing requirements could be justified if data sharing is a part of good scientific practice.

Ten simple rules for starting FAIR discussions in your community

This work presents 10 rules that provide guidance and recommendations on how to start up discussions around the implementation of the FAIR (Findable, Accessible, Interoperable, Reusable) principles and creation of standardised ways of working. These recommendations will be particularly relevant if you are unsure where to start, who to involve, what the benefits and barriers of standardisation are, and if little work has been done in your discipline to standardise research workflows. When applied, these rules will support a more effective way of engaging the community with discussions on standardisation and practical implementation of the FAIR principles.

Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study

OBJECTIVE

Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics.

MATERIALS AND METHODS

Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics.

RESULTS

On average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5.

CONCLUSIONS

Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.

Neither carrots nor sticks? Challenges surrounding data sharing from the perspective of research funding agencies—A qualitative expert interview study

Background

Data Sharing is widely recognised as crucial for accelerating scientific research and improving its quality. However, data sharing is still not a common practice. Funding agencies tend to facilitate the sharing of research data by both providing incentives and requiring data sharing as part of their policies and conditions for awarding grants. The goal of our article is to answer the following question: What challenges do international funding agencies see when it comes to their own efforts to foster and implement data sharing through their policies?

Methods

We conducted a series of sixteen guideline-based expert interviews with representatives of leading international funding agencies. As contact persons for open science at their respective agencies, they offered their perspectives and experiences concerning their organisations’ data sharing policies. We performed a qualitative content analysis of the interviews and categorised the challenges perceived by funding agencies.

Results

We identify and illustrate six challenges surrounding data sharing policies as perceived by leading funding agencies: The design of clear policies, monitoring of compliance, sanctions for non-compliance, incentives, support, and limitations for funders’ own capabilities. However, our interviews also show how funders approach potential solutions to overcome these challenges, for example by coordinating with other agencies or adjusting grant evaluation metrics to incentivise data sharing.

Discussion and conclusion

Our interviews point to existing flaws in funders’ data sharing policies, such as a lack of clarity, a lack of monitoring of funded researchers’ data sharing behaviour, and a lack of incentives. A number of agencies could suggest potential solutions but often struggle with the overall complexity of data sharing and the implementation of these measures. Funders cannot solve each challenge by themselves, but they can play an active role and lead joint efforts towards a culture of data sharing.

The challenges of data in future pandemics

The use of data has been essential throughout the unfolding COVID-19 pandemic. We have needed it to populate our models, inform our understanding, and shape our responses to the disease. However, data has not always been easy to find and access, it has varied in quality and coverage, been difficult to reuse or repurpose. This paper reviews these and other challenges and recommends steps to develop a data ecosystem better able to deal with future pandemics by better supporting preparedness, prevention, detection and response.

Translating research into health practice: a case study of integrated palliative care system in Catalonia, Spain

Purpose

Research findings provide the professional community with knowledge that enables to better understand healthcare interventions. Many authors point out that whilst these findings are valued, the findings are not always translated into healthcare practise. The purpose of the paper is to assess the applicability of the essential elements of an integrated palliative care system (IPCS) found in research into the practise of Osona Palliative Care System (OPCS).

Design/methodology/approach

The study used a qualitative methodology with a case study design. In total, 24 health professionals were interviewed in Osona for the research, and the results were analysed using deductive content analysis.

Findings

The study concludes that research findings can better be translated into specific contexts by incorporating the needs and characteristics of the system. The process could be a strategy for bridging the research–practise gap.

Originality/value

Combining the findings from the study and the findings found in the literature reviewed led to the creation of the IPCS-elements-blended model of research and practise. Such a kind of mixed model could be used in other studies seeking to overcome the research and practice gap.

Primer on binary logistic regression

Family medicine has traditionally prioritised patient care over research. However, recent recommendations to strengthen family medicine include calls to focus more on research including improving research methods used in the field. Binary logistic regression is one method frequently used in family medicine research to classify, explain or predict the values of some characteristic, behaviour or outcome. The binary logistic regression model relies on assumptions including independent observations, no perfect multicollinearity and linearity. The model produces ORs, which suggest increased, decreased or no change in odds of being in one category of the outcome with an increase in the value of the predictor. Model significance quantifies whether the model is better than the baseline value (ie, the percentage of people with the outcome) at explaining or predicting whether the observed cases in the data set have the outcome. One model fit measure is the count- R2, which is the percentage of observations where the model correctly predicted the outcome variable value. Related to the count- R2 are model sensitivity—the percentage of those with the outcome who were correctly predicted to have the outcome—and specificity—the percentage of those without the outcome who were correctly predicted to not have the outcome. Complete model reporting for binary logistic regression includes descriptive statistics, a statement on whether assumptions were checked and met, ORs and CIs for each predictor, overall model significance and overall model fit.

The new normal? Redaction bias in biomedical science

A concerning amount of biomedical research is not reproducible. Unreliable results impede empirical progress in medical science, ultimately putting patients at risk. Many proximal causes of this irreproducibility have been identified, a major one being inappropriate statistical methods and analytical choices by investigators. Within this, we formally quantify the impact of inappropriate redaction beyond a threshold value in biomedical science. This is effectively truncation of a dataset by removing extreme data points, and we elucidate its potential to accidentally or deliberately engineer a spurious result in significance testing. We demonstrate that the removal of a surprisingly small number of data points can be used to dramatically alter a result. It is unknown how often redaction bias occurs in the broader literature, but given the risk of distortion to the literature involved, we suggest that it must be studiously avoided, and mitigated with approaches to counteract any potential malign effects to the research quality of medical science.

Rates and predictors of data and code sharing in the medical and health sciences: Protocol for a systematic review and individual participant data meta-analysis

Numerous studies have demonstrated low but increasing rates of data and code sharing within medical and health research disciplines. However, it remains unclear how commonly data and code are shared across all fields of medical and health research, as well as whether sharing rates are positively associated with implementation of progressive policies by publishers and funders, or growing expectations from the medical and health research community at large. Therefore this systematic review aims to synthesise the findings of medical and health science studies that have empirically investigated the prevalence of data or code sharing, or both. Objectives include the investigation of: (i) the prevalence of public sharing of research data and code alongside published articles (including preprints), (ii) the prevalence of private sharing of research data and code in response to reasonable requests, and (iii) factors associated with the sharing of either research output (e.g., the year published, the publisher's policy on sharing, the presence of a data or code availability statement). It is hoped that the results will provide some insight into how often research data and code are shared publicly and privately, how this has changed over time, and how effective some measures such as the institution of data sharing policies and data availability statements have been in motivating researchers to share their underlying data and code.

Quantifying the influence of bias in reproductive and perinatal epidemiology through simulation

PURPOSE

The application of simulated data in epidemiological studies enables the illustration and quantification of the magnitude of various types of bias commonly found in observational studies. This was a review of the application of simulation methods to the quantification of bias in reproductive and perinatal epidemiology and an assessment of value gained.

METHODS

A search of published studies available in English was conducted in August 2020 using PubMed, Medline, Embase, CINAHL, and Scopus. A gray literature search of Google and Google Scholar, and a hand search using the reference lists of included studies was undertaken.

RESULTS

Thirty-nine papers were included in this study, covering information (n = 14), selection (n = 14), confounding (n = 9), protection (n = 1), and attenuation bias (n = 1). The methods of simulating data and reporting of results varied, with more recent studies including causal diagrams. Few studies included code for replication.

CONCLUSIONS

Although there has been an increasing application of simulation in reproductive and perinatal epidemiology since 2015, overall this remains an underexplored area. Further efforts are required to increase knowledge of how the application of simulation can quantify the influence of bias, including improved design, analysis and reporting. This will improve causal interpretation in reproductive and perinatal studies.

Understanding experiments and research practices for reproducibility: an exploratory study

Scientific experiments and research practices vary across disciplines. The research practices followed by scientists in each domain play an essential role in the understandability and reproducibility of results. The "Reproducibility Crisis", where researchers find difficulty in reproducing published results, is currently faced by several disciplines. To understand the underlying problem in the context of the reproducibility crisis, it is important to first know the different research practices followed in their domain and the factors that hinder reproducibility. We performed an exploratory study by conducting a survey addressed to researchers representing a range of disciplines to understand scientific experiments and research practices for reproducibility. The survey findings identify a reproducibility crisis and a strong need for sharing data, code, methods, steps, and negative and positive results. Insufficient metadata, lack of publicly available data, and incomplete information in study methods are considered to be the main reasons for poor reproducibility. The survey results also address a wide number of research questions on the reproducibility of scientific results. Based on the results of our explorative study and supported by the existing published literature, we offer general recommendations that could help the scientific community to understand, reproduce, and reuse experimental data and results in the research data lifecycle.

Decoding Neuromuscular Disorders Using Phenotypic Clusters Obtained From Co-Occurrence Networks

Neuromuscular disorders (NMDs) represent an important subset of rare diseases associated with elevated morbidity and mortality whose diagnosis can take years. Here we present a novel approach using systems biology to produce functionally-coherent phenotype clusters that provide insight into the cellular functions and phenotypic patterns underlying NMDs, using the Human Phenotype Ontology as a common framework. Gene and phenotype information was obtained for 424 NMDs in OMIM and 126 NMDs in Orphanet, and 335 and 216 phenotypes were identified as typical for NMDs, respectively. ‘Elevated serum creatine kinase’ was the most specific to NMDs, in agreement with the clinical test of elevated serum creatinine kinase that is conducted on NMD patients. The approach to obtain co-occurring NMD phenotypes was validated based on co-mention in PubMed abstracts. A total of 231 (OMIM) and 150 (Orphanet) clusters of highly connected co-occurrent NMD phenotypes were obtained. In parallel, a tripartite network based on phenotypes, diseases and genes was used to associate NMD phenotypes with functions, an approach also validated by literature co-mention, with KEGG pathways showing proportionally higher overlap than Gene Ontology and Reactome. Phenotype-function pairs were crossed with the co-occurrent NMD phenotype clusters to obtain 40 (OMIM) and 72 (Orphanet) functionally coherent phenotype clusters. As expected, many of these overlapped with known diseases and confirmed existing knowledge. Other clusters revealed interesting new findings, indicating informative phenotypes for differential diagnosis, providing deeper knowledge of NMDs, and pointing towards specific cell dysfunction caused by pleiotropic genes. This work is an example of reproducible research that i) can help better understand NMDs and support their diagnosis by providing a new tool that exploits existing information to obtain novel clusters of functionally-related phenotypes, and ii) takes us another step towards personalised medicine for NMDs.

Reproducible and transparent research practices in published neurology research

BACKGROUND

The objective of this study was to evaluate the nature and extent of reproducible and transparent research practices in neurology publications.

METHODS

The NLM catalog was used to identify MEDLINE-indexed neurology journals. A PubMed search of these journals was conducted to retrieve publications over a 5-year period from 2014 to 2018. A random sample of publications was extracted. Two authors conducted data extraction in a blinded, duplicate fashion using a pilot-tested Google form. This form prompted data extractors to determine whether publications provided access to items such as study materials, raw data, analysis scripts, and protocols. In addition, we determined if the publication was included in a replication study or systematic review, was preregistered, had a conflict of interest declaration, specified funding sources, and was open access.

RESULTS

Our search identified 223,932 publications meeting the inclusion criteria, from which 400 were randomly sampled. Only 389 articles were accessible, yielding 271 publications with empirical data for analysis. Our results indicate that 9.4% provided access to materials, 9.2% provided access to raw data, 0.7% provided access to the analysis scripts, 0.7% linked the protocol, and 3.7% were preregistered. A third of sampled publications lacked funding or conflict of interest statements. No publications from our sample were included in replication studies, but a fifth were cited in a systematic review or meta-analysis.

CONCLUSIONS

Currently, published neurology research does not consistently provide information needed for reproducibility. The implications of poor research reporting can both affect patient care and increase research waste. Collaborative intervention by authors, peer reviewers, journals, and funding sources is needed to mitigate this problem.

Do oncology researchers adhere to reproducible and transparent principles? A cross-sectional survey of published oncology literature

OBJECTIVES

As much as 50%-90% of research is estimated to be irreproducible, costing upwards of $28 billion in USA alone. Reproducible research practices are essential to improving the reproducibility and transparency of biomedical research, such as including preregistering studies, publishing a protocol, making research data and metadata publicly available, and publishing in open access journals. Here we report an investigation of key reproducible or transparent research practices in the published oncology literature.

DESIGN

We performed a cross-sectional analysis of a random sample of 300 oncology publications published from 2014 to 2018. We extracted key reproducibility and transparency characteristics in a duplicative fashion by blinded investigators using a pilot tested Google Form.

PRIMARY OUTCOME MEASURES

The primary outcome of this investigation is the frequency of key reproducible or transparent research practices followed in published biomedical and clinical oncology literature.

RESULTS

Of the 300 publications randomly sampled, 296 were analysed for reproducibility characteristics. Of these 296 publications, 194 contained empirical data that could be analysed for reproducible and transparent research practices. Raw data were available for nine studies (4.6%). Five publications (2.6%) provided a protocol. Despite our sample including 15 clinical trials and 7 systematic reviews/meta-analyses, only 7 included a preregistration statement. Less than 25% (65/194) of publications provided an author conflict of interest statement.

CONCLUSION

We found that key reproducibility and transparency characteristics were absent from a random sample of published oncology publications. We recommend required preregistration for all eligible trials and systematic reviews, published protocols for all manuscripts, and deposition of raw data and metadata in public repositories.