Reproducible Research Practices and Transparency across the Biomedical Literature

Measuring the replicability of our own research

In the study of transgenic mouse models of neurodevelopmental and neurodegenerative disorders, we use batteries of tests to measure deficits in behaviour and from the results of these tests, we make inferences about the mental states of the mice that we interpret as deficits in "learning", "memory", "anxiety", "depression", etc. This paper discusses the problems of determining whether a particular transgenic mouse is a valid mouse model of disease X, the problem of background strains, and the question of whether our behavioural tests are measuring what we say they are. The problem of the reliability of results is then discussed: are they replicable between labs and can we replicate our results in our own lab? This involves the study of intra- and inter- experimenter reliability. The variables that influence replicability and the importance of conducting a complete behavioural phenotype: sensory, motor, cognitive and social emotional behaviour are discussed. Then the thorny question of failure to replicate is examined: Is it a curse or a blessing? Finally, the role of failure in research and what it tells us about our research paradigms is examined.

Assessment of transparency indicators in space medicine

Space medicine is a vital discipline with often time-intensive and costly projects and constrained opportunities for studying various elements such as space missions, astronauts, and simulated environments. Moreover, private interests gain increasing influence in this discipline. In scientific disciplines with these features, transparent and rigorous methods are essential. Here, we undertook an evaluation of transparency indicators in publications within the field of space medicine. A meta-epidemiological assessment of PubMed Central Open Access (PMC OA) eligible articles within the field of space medicine was performed for prevalence of code sharing, data sharing, pre-registration, conflicts of interest, and funding. Text mining was performed with the rtransparent text mining algorithms with manual validation of 200 random articles to obtain corrected estimates. Across 1215 included articles, 39 (3%) shared code, 258 (21%) shared data, 10 (1%) were registered, 110 (90%) contained a conflict-of-interest statement, and 1141 (93%) included a funding statement. After manual validation, the corrected estimates for code sharing, data sharing, and registration were 5%, 27%, and 1%, respectively. Data sharing was 32% when limited to original articles and highest in space/parabolic flights (46%). Overall, across space medicine we observed modest rates of data sharing, rare sharing of code and almost non-existent protocol registration. Enhancing transparency in space medicine research is imperative for safeguarding its scientific rigor and reproducibility.

CONSORT-TM: Text classification models for assessing the completeness of randomized controlled trial publications

Objective

To develop text classification models for determining whether the checklist items in the CONSORT reporting guidelines are reported in randomized controlled trial publications.

Materials and Methods

Using a corpus annotated at the sentence level with 37 fine-grained CONSORT items, we trained several sentence classification models (PubMedBERT fine-tuning, BioGPT fine-tuning, and in-context learning with GPT-4) and compared their performance. To address the problem of small training dataset, we used several data augmentation methods (EDA, UMLS-EDA, text generation and rephrasing with GPT-4) and assessed their impact on the fine-tuned PubMedBERT model. We also fine-tuned PubMedBERT models limited to checklist items associated with specific sections (e.g., Methods) to evaluate whether such models could improve performance compared to the single full model. We performed 5-fold cross-validation and report precision, recall, F1 score, and area under curve (AUC).

Results

Fine-tuned PubMedBERT model that takes as input the sentence and the surrounding sentence representations and uses section headers yielded the best overall performance (0.71 micro-F1, 0.64 macro-F1). Data augmentation had limited positive effect, UMLS-EDA yielding slightly better results than data augmentation using GPT-4. BioGPT fine-tuning and GPT-4 in-context learning exhibited suboptimal results. Methods-specific model yielded higher performance for methodology items, other section-specific models did not have significant impact.

Conclusion

Most CONSORT checklist items can be recognized reasonably well with the fine-tuned PubMedBERT model but there is room for improvement. Improved models can underpin the journal editorial workflows and CONSORT adherence checks and can help authors in improving the reporting quality and completeness of their manuscripts.

Advocating for Generalizability: Accepting Inherent Variability in Translation of Animal Research Outcomes

Advancing scientific discovery requires investigators to embrace research practices that increase transparency and disclosure about materials, methods, and outcomes. Several research advocacy and funding organizations have produced guidelines and recommended practices to enhance reproducibility through detailed and rigorous research approaches; however, confusion around vocabulary terms and a lack of adoption of suggested practices have stymied successful implementation. Although reproducibility of research findings cannot be guaranteed due to extensive inherent variables in attempts at experimental repetition, the scientific community can advocate for generalizability in the application of data outcomes to ensure a broad and effective impact on the comparison of animals to translation within human research. This report reviews suggestions, based upon work with National Institutes of Health advisory groups, for improving rigor and transparency in animal research through aspects of experimental design, statistical assessment, and reporting factors to advocate for generalizability in the application of comparative outcomes between animals and humans.

Up Front and Open? Shrouded in Secrecy? Or Somewhere in Between? A Meta-Research Systematic Review of Open Science Practices in Sport Medicine Research

OBJECTIVE: To investigate open science practices in research published in the top 5 sports medicine journals from May 1, 2022, and October 1, 2022. DESIGN: A meta-research systematic review. LITERATURE SEARCH: Open science practices were searched in MEDLINE. STUDY SELECTION CRITERIA: We included original scientific research published in one of the identified top 5 sports medicine journals in 2022 as ranked by Clarivate: (1) British Journal of Sports Medicine, (2) Journal of Sport and Health Science, (3) American Journal of Sports Medicine, (4) Medicine and Science in Sports and Exercise, and (5) Sports Medicine-Open. Studies were excluded if they were systematic reviews, qualitative research, gray literature, or animal or cadaver models. DATA SYNTHESIS: Open science practices were extracted in accordance with the Transparency and Openness Promotion guidelines and patient and public involvement. RESULTS: Two hundred forty-three studies were included. The median number of open science practices in each study was 2, out of a maximum of 12 (range: 0-8; interquartile range: 2). Two hundred thirty-four studies (96%, 95% confidence interval [CI]: 94%-99%) provided an author conflict-of-interest statement and 163 (67%, 95% CI: 62%-73%) reported funding. Twenty-one studies (9%, 95% CI: 5%-12%) provided open-access data. Fifty-four studies (22%, 95% CI: 17%-27%) included a data availability statement and 3 (1%, 95% CI: 0%-3%) made code available. Seventy-six studies (32%, 95% CI: 25%-37%) had transparent materials and 30 (12%, 95% CI: 8%-16%) used a reporting guideline. Twenty-eight studies (12%, 95% CI: 8%-16%) were preregistered. Six studies (3%, 95% CI: 1%-4%) published a protocol. Four studies (2%, 95% CI: 0%-3%) reported an analysis plan a priori. Seven studies (3%, 95% CI: 1%-5%) reported patient and public involvement. CONCLUSION: Open science practices in the sports medicine field are extremely limited. The least followed practices were sharing code, data, and analysis plans. J Orthop Sports Phys Ther 2023;53(12):1-13. Epub 20 October 2023. doi:10.2519/jospt.2023.12016.

Experience of irreproducibility as a risk factor for poor mental health in biomedical science doctoral students: A survey and interview-based study

High rates of irreproducibility and of poor mental health in graduate students have been reported in the biomedical sciences in the past ten years, but to date, little research has investigated whether these two trends interact. In this study, we ask whether the experience of failing to replicate an expected finding impacts graduate students' mental health. Using an online survey paired with semi-structured qualitative interviews, we examined how often biomedical science doctoral students at a large American public university experienced events that could be interpreted as failures to replicate and how they responded to these experiences. We found that almost all participants had experience with irreproducibility: 84% had failed to replicate their own results, 70% had failed to replicate a colleague's finding, and 58% had failed to replicate a result from the published literature. Participants reported feelings of self-doubt, frustration, and depression while experiencing irreproducibility, and in 24% of cases, these emotional responses were strong enough to interfere with participants' eating, sleeping, or ability to work. A majority (82%) of participants initially believed that the anomalous results could be attributed to their own error. However, after further experimentation, most participants concluded that the original result was wrong (38%), that there was a key difference between the original experiment and their own (17%), or that there was a problem with the protocol (17%). These results suggest that biomedical science graduate students may be biased towards initially interpreting failures to replicate as indicative of a lack of skill, which may trigger or perpetuate feelings of anxiety, depression, or impostorism.

Shift in hospital opioid use during the COVID-19 pandemic in Brazil: a time-series analysis of one million prescriptions

The pronounced change in the profile of hospitalized patients during COVID-19 and the severe respiratory component of this disease, with a great need for mechanical ventilation, led to changes in the consumption pattern of some medicines and supplies. This time-series study analyzed the in-hospital consumption of opioids during the COVID-19 pandemic in 24 Brazilian hospitals compared to the pre-pandemic period. Data included 711,883 adult patients who had opioids prescribed. In 2020, the mean consumption was significantly higher compared to 2019 for parenteral fentanyl, enteral methadone, and parenteral methadone. It was significantly lower for parenteral morphine parenteral sufentanil, and parenteral tramadol. For remifentanil, it did not differ. The number of patients in 2020 was lower but the mean consumption was higher for fentanyl, parenteral methadone, and remifentanil. It was lower for enteral methadone and parenteral sufentanil. The consumption of parenteral morphine and parenteral tramadol was stable. There was a relevant increase in hospital consumption of some potent opioids during the COVID-19 pandemic in Brazil. These results reinforce the concern about epidemiological surveillance of opioid use after periods of increased hospital use since in-hospital consumption can be the gateway to the misuse or other than the prescribed use of opioids after discharge.

Few randomized trials in preterm birth prevention meet predefined usefulness criteria

OBJECTIVES

We operationalized a research usefulness tool identified through literature searches and consensus and examined if randomized controlled trials (RCTs) addressing preterm birth prevention met predefined criteria for usefulness.

STUDY DESIGN AND SETTING

The usefulness tool included eight criteria combining 13 items. RCTs were evaluated for compliance with each item by multiple assessors (reviewer agreement 95-98%). Proportions of compliances with 95% confidence interval (CI) were calculated and change over time was assessed using ≧ 2010 as a cutoff.

RESULTS

Among 347 selected RCTs, published within 56 preterm birth Cochrane reviews, only 36 (10%, 95% CI = 7-14%) met more than half of the usefulness criteria. Compared to trials before 2010, recent trials used composite or surrogate (less informative) outcomes more often (13% vs. 25%, relative risk 1.91, 95% CI = 1.21-3.00). Only 16 trials reflected real practice (pragmatism) in design (5%, 95% CI = 3-7%), with no improvements over time. No trials reported involvement of mothers to reflect patients' research priorities and outcomes selection. Recent trials were more transparent.

CONCLUSION

Few preterm birth prevention RCTs met more than half of the usefulness criteria but most of usefulness criteria are improving after 2010. Use of informative outcomes, patient centeredness, pragmatism and transparency should be key targets for future research planning.

Transparency in Artificial Intelligence Research: a Systematic Review of Availability Items Related to Open Science in Radiology and Nuclear Medicine

RATIONALE AND OBJECTIVES

Reproducibility of artificial intelligence (AI) research has become a growing concern. One of the fundamental reasons is the lack of transparency in data, code, and model. In this work, we aimed to systematically review the radiology and nuclear medicine papers on AI in terms of transparency and open science.

MATERIALS AND METHODS

A systematic literature search was performed in PubMed to identify original research studies on AI. The search was restricted to studies published in Q1 and Q2 journals that are also indexed on the Web of Science. A random sampling of the literature was performed. Besides six baseline study characteristics, a total of five availability items were evaluated. Two groups of independent readers including eight readers participated in the study. Inter-rater agreement was analyzed. Disagreements were resolved with consensus.

RESULTS

Following eligibility criteria, we included a final set of 194 papers. The raw data was available in about one-fifth of the papers (34/194; 18%). However, the authors made their private data available only in one paper (1/161; 1%). About one-tenth of the papers made their pre-modeling (25/194; 13%), modeling (28/194; 14%), or post-modeling files (15/194; 8%) available. Most of the papers (189/194; 97%) did not attempt to create a ready-to-use system for real-world usage. Data origin, use of deep learning, and external validation had statistically significantly different distributions. The use of private data alone was negatively associated with the availability of at least one item (p<0.001).

CONCLUSION

Overall rates of availability for items were poor, leaving room for substantial improvement.

Methodology reporting improved over time in 176,469 randomized controlled trials

OBJECTIVES

To describe randomized controlled trial (RCT) methodology reporting over time.

STUDY DESIGN AND SETTING

We used a deep learning-based sentence classification model based on the Consolidated Standards of Reporting Trials (CONSORT) statement, considered minimum requirements for reporting RCTs. We included 176,469 RCT reports published between 1966 and 2018. We analyzed the reporting trends over 5-year time periods, grouping trials from 1966 to 1990 in a single stratum. We also explored the effect of journal impact factor (JIF) and medical discipline.

RESULTS

Population, Intervention, Comparator, Outcome (PICO) items were commonly reported during each period, and reporting increased over time (e.g., interventions: 79.1% during 1966-1990 to 87.5% during 2010-2018). Reporting of some methods information has increased, although there is room for improvement (e.g., sequence generation: 10.8-41.8%). Some items are reported infrequently (e.g., allocation concealment: 5.1-19.3%). The number of items reported and JIF are weakly correlated (Pearson's r (162,702) = 0.16, P < 0.001). The differences in the proportion of items reported between disciplines are small (<10%).

CONCLUSION

Our analysis provides large-scale quantitative support for the hypothesis that RCT methodology reporting has improved over time. Extending these models to all CONSORT items could facilitate compliance checking during manuscript authoring and peer review, and support metaresearch.

Evaluating the Utility and Privacy of Synthetic Breast Cancer Clinical Trial Data Sets

PURPOSE

There is strong interest from patients, researchers, the pharmaceutical industry, medical journal editors, funders of research, and regulators in sharing clinical trial data for secondary analysis. However, data access remains a challenge because of concerns about patient privacy. It has been argued that synthetic data generation (SDG) is an effective way to address these privacy concerns. There is a dearth of evidence supporting this on oncology clinical trial data sets, and on the utility of privacy-preserving synthetic data. The objective of the proposed study is to validate the utility and privacy risks of synthetic clinical trial data sets across multiple SDG techniques.

METHODS

We synthesized data sets from eight breast cancer clinical trial data sets using three types of generative models: sequential synthesis, conditional generative adversarial network, and variational autoencoder. Synthetic data utility was evaluated by replicating the published analyses on the synthetic data and assessing concordance of effect estimates and CIs between real and synthetic data. Privacy was evaluated by measuring attribution disclosure risk and membership disclosure risk.

RESULTS

Utility was highest using the sequential synthesis method where all results were replicable and the CI overlap most similar or higher for seven of eight data sets. Both types of privacy risks were low across all three types of generative models.

DISCUSSION

Synthetic data using sequential synthesis methods can act as a proxy for real clinical trial data sets, and simultaneously have low privacy risks. This type of generative model can be one way to enable broader sharing of clinical trial data.

Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals

BACKGROUND

Infectious diseases carry large global burdens and have implications for society at large. Therefore, reproducible, transparent research is extremely important.

METHODS

We evaluated transparency indicators (code and data sharing, registration, and conflict and funding disclosures) in the 5340 PubMed Central Open Access articles published in 2019 or 2021 in the 9 most cited specialty journals in infectious diseases using the text-mining R package, rtransparent.

RESULTS

A total of 5340 articles were evaluated (1860 published in 2019 and 3480 in 2021 [of which 1828 were on coronavirus disease 2019, or COVID-19]). Text mining identified code sharing in 98 (2%) articles, data sharing in 498 (9%), registration in 446 (8%), conflict of interest disclosures in 4209 (79%), and funding disclosures in 4866 (91%). There were substantial differences across the 9 journals: 1%-9% for code sharing, 5%-25% for data sharing, 1%-31% for registration, 7%-100% for conflicts of interest, and 65%-100% for funding disclosures. Validation-corrected imputed estimates were 3%, 11%, 8%, 79%, and 92%, respectively. There were no major differences between articles published in 2019 and non-COVID-19 articles in 2021. In 2021, non-COVID-19 articles had more data sharing (12%) than COVID-19 articles (4%).

CONCLUSIONS

Data sharing, code sharing, and registration are very uncommon in infectious disease specialty journals. Increased transparency is required.

Comparison of Reporting and Transparency in Published Protocols and Publications in Umbrella Reviews: Scoping Review

BACKGROUND

Inconsistencies between a protocol and its umbrella review (UR) may mislead readers about the importance of findings or lead to false-positive results. Furthermore, not documenting and explaining inconsistencies in the UR could reduce its transparency. To our knowledge, no study has examined the methodological consistency of the protocols with their URs and assessed the transparency of the URs when generating evidence.

OBJECTIVE

This study aimed to investigate the inconsistency of protocols with their URs in the methodology and assess the transparency of the URs.

METHODS

We searched medical-related electronic databases from their inception to January 1, 2022. We investigated inconsistencies between protocols and their publications and transparencies in the search strategy, inclusion criteria, methods of screening and data extraction, quality assessment, and statistical analysis.

RESULTS

We included 31 protocols and 35 publications. For the search strategy, 39 inconsistencies between the protocols and their publications were found in 26 of the 35 (74%) URs, and 16 of these inconsistencies were indicated and explained. There were 84 inconsistencies between the protocols and their URs regarding the inclusion criteria in 31 of the 35 (89%) URs, and 29 of the inconsistencies were indicated and explained. Deviations from their protocols were found in 12 of the 32 (38%) URs reporting the methods of screening, 14 of the 30 (47%) URs reporting the methods of data extraction, and 11 of the 32 (34%) URs reporting the methods for quality assessment. Of the 35 URs, 6 (17%) were inconsistent with their protocols in terms of the tools for quality assessment; one-half (3/6, 50%) of them indicated and explained the deviations. As for the statistical analysis, 31 of the 35 (89%) URs generated 61 inconsistencies between the publications and their protocols, and 16 inconsistencies were indicated and explained.

CONCLUSIONS

There was a high prevalence of inconsistencies between protocols and publications of URs, and more than one-half of the inconsistencies were not indicated and explained in the publications. Therefore, how to promote the transparency of URs will be a major part of future work.

Prevalence and predictors of data and code sharing in the medical and health sciences: systematic review with meta-analysis of individual participant data

OBJECTIVES

To synthesise research investigating data and code sharing in medicine and health to establish an accurate representation of the prevalence of sharing, how this frequency has changed over time, and what factors influence availability.

DESIGN

Systematic review with meta-analysis of individual participant data.

DATA SOURCES

Ovid Medline, Ovid Embase, and the preprint servers medRxiv, bioRxiv, and MetaArXiv were searched from inception to 1 July 2021. Forward citation searches were also performed on 30 August 2022.

REVIEW METHODS

Meta-research studies that investigated data or code sharing across a sample of scientific articles presenting original medical and health research were identified. Two authors screened records, assessed the risk of bias, and extracted summary data from study reports when individual participant data could not be retrieved. Key outcomes of interest were the prevalence of statements that declared that data or code were publicly or privately available (declared availability) and the success rates of retrieving these products (actual availability). The associations between data and code availability and several factors (eg, journal policy, type of data, trial design, and human participants) were also examined. A two stage approach to meta-analysis of individual participant data was performed, with proportions and risk ratios pooled with the Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis.

RESULTS

The review included 105 meta-research studies examining 2 121 580 articles across 31 specialties. Eligible studies examined a median of 195 primary articles (interquartile range 113-475), with a median publication year of 2015 (interquartile range 2012-2018). Only eight studies (8%) were classified as having a low risk of bias. Meta-analyses showed a prevalence of declared and actual public data availability of 8% (95% confidence interval 5% to 11%) and 2% (1% to 3%), respectively, between 2016 and 2021. For public code sharing, both the prevalence of declared and actual availability were estimated to be <0.5% since 2016. Meta-regressions indicated that only declared public data sharing prevalence estimates have increased over time. Compliance with mandatory data sharing policies ranged from 0% to 100% across journals and varied by type of data. In contrast, success in privately obtaining data and code from authors historically ranged between 0% and 37% and 0% and 23%, respectively.

CONCLUSIONS

The review found that public code sharing was persistently low across medical research. Declarations of data sharing were also low, increasing over time, but did not always correspond to actual sharing of data. The effectiveness of mandatory data sharing policies varied substantially by journal and type of data, a finding that might be informative for policy makers when designing policies and allocating resources to audit compliance.

SYSTEMATIC REVIEW REGISTRATION

Open Science Framework doi:10.17605/OSF.IO/7SX8U.

Openness and Computational Reproducibility in Plant Pathology: Where We Stand and a Way Forward

Open research practices have been highlighted extensively during the last 10 years in many fields of scientific study as essential standards needed to promote transparency and reproducibility of scientific results. Scientific claims can only be evaluated based on how protocols, materials, equipment, and methods were described; data were collected and prepared; and analyses were conducted. Openly sharing protocols, data, and computational code is central to current scholarly dissemination and communication, but in many fields, including plant pathology, adoption of these practices has been slow. We randomly selected 450 articles published from 2012 to 2021 across 21 journals representative of the plant pathology discipline and assigned them scores reflecting their openness and computational reproducibility. We found that most of the articles did not follow protocols for open science and failed to share data or code in a reproducible way. We propose that use of open-source tools facilitates computationally reproducible work and analyses, benefitting not just readers but the authors as well. Finally, we provide ideas and suggest tools to promote open, reproducible computational research practices among plant pathologists. [Formula: see text] Copyright © 2023 The Author(s). This is an open access article distributed under the CC BY 4.0 International license.

Biomedical supervisors' role modeling of open science practices

Supervision is one important way to socialize Ph.D. candidates into open and responsible research. We hypothesized that one should be more likely to identify open science practices (here publishing open access and sharing data) in empirical publications that were part of a Ph.D. thesis when the Ph.D. candidates' supervisors engaged in these practices compared to those whose supervisors did not or less often did. Departing from thesis repositories at four Dutch University Medical centers, we included 211 pairs of supervisors and Ph.D. candidates, resulting in a sample of 2062 publications. We determined open access status using UnpaywallR and Open Data using Oddpub, where we also manually screened publications with potential open data statements. Eighty-three percent of our sample was published openly, and 9% had open data statements. Having a supervisor who published open access more often than the national average was associated with an odds of 1.99 to publish open access. However, this effect became nonsignificant when correcting for institutions. Having a supervisor who shared data was associated with 2.22 (CI:1.19-4.12) times the odds to share data compared to having a supervisor that did not. This odds ratio increased to 4.6 (CI:1.86-11.35) after removing false positives. The prevalence of open data in our sample was comparable to international studies; open access rates were higher. Whilst Ph.D. candidates spearhead initiatives to promote open science, this study adds value by investigating the role of supervisors in promoting open science.

Partial Identification of Personalized Treatment Response with Trial-reported Analyses of Binary Subgroups

Medical journals have adhered to a reporting practice that seriously limits the usefulness of published trial findings. Medical decision makers commonly observe many patient covariates and seek to use this information to personalize treatment choices. Yet standard summaries of trial findings only partition subjects into broad subgroups, typically binary categories. Given this reporting practice, we study the problem of inference on long mean treatment outcomes E[y(t)|x], where t is a treatment, y(t) is a treatment outcome, and the covariate vector x has length K, each component being a binary variable. The available data are estimates of {E[y(t)|x k = 0], E[y(t)|x k = 1], P(x k )}, k = 1,..., K reported in journal articles. We show that reported trial findings partially identify {E[y(t)|x], P(x)}. Illustrative computations demonstrate that the summaries of trial findings in journal articles may imply only wide bounds on long mean outcomes. One can realistically tighten inferences if one can combine reported trial findings with credible assumptions having identifying power, such as bounded-variation assumptions.

Trends in the sample size, statistics, and contributions to the BrainMap database of activation likelihood estimation meta-analyses: An empirical study of 10-year data

The literature of neuroimaging meta-analysis has been thriving for over a decade. A majority of them were coordinate-based meta-analyses, particularly the activation likelihood estimation (ALE) approach. A meta-evaluation of these meta-analyses was performed to qualitatively evaluate their design and reporting standards. The publications listed from the BrainMap website were screened. Six hundred and three ALE papers published during 2010-2019 were included and analysed. For reporting standards, most of the ALE papers reported their total number of Papers involved and mentioned the inclusion/exclusion criteria on Paper selection. However, most papers did not describe how data redundancy was avoided when multiple related Experiments were reported within one paper. The most prevalent repeated-measures correction methods were voxel-level FDR (54.4%) and cluster-level FWE (33.8%), with the latter quickly replacing the former since 2016. For study characteristics, sample size in terms of number of Papers included per ALE paper and number of Experiments per analysis seemed to be stable over the decade. One-fifth of the surveyed ALE papers failed to meet the recommendation of having >17 Experiments per analysis. For data sharing, most of them did not provide input and output data. In conclusion, the field has matured well in terms of rising dominance of cluster-level FWE correction, and slightly improved reporting on elimination of data redundancy and providing input data. The provision of Data and Code availability statements and flow chart of literature screening process, as well as data submission to BrainMap, should be more encouraged.

The logical structure of experiments lays the foundation for a theory of reproducibility

The scientific reform movement has proposed openness as a potential remedy to the putative reproducibility or replication crisis. However, the conceptual relationship among openness, replication experiments and results reproducibility has been obscure. We analyse the logical structure of experiments, define the mathematical notion of idealized experiment and use this notion to advance a theory of reproducibility. Idealized experiments clearly delineate the concepts of replication and results reproducibility, and capture key differences with precision, allowing us to study the relationship among them. We show how results reproducibility varies as a function of the elements of an idealized experiment, the true data-generating mechanism, and the closeness of the replication experiment to an original experiment. We clarify how openness of experiments is related to designing informative replication experiments and to obtaining reproducible results. With formal backing and evidence, we argue that the current 'crisis' reflects inadequate attention to a theoretical understanding of results reproducibility.

EnzymeML: seamless data flow and modeling of enzymatic data

The design of biocatalytic reaction systems is highly complex owing to the dependency of the estimated kinetic parameters on the enzyme, the reaction conditions, and the modeling method. Consequently, reproducibility of enzymatic experiments and reusability of enzymatic data are challenging. We developed the XML-based markup language EnzymeML to enable storage and exchange of enzymatic data such as reaction conditions, the time course of the substrate and the product, kinetic parameters and the kinetic model, thus making enzymatic data findable, accessible, interoperable and reusable (FAIR). The feasibility and usefulness of the EnzymeML toolbox is demonstrated in six scenarios, for which data and metadata of different enzymatic reactions are collected and analyzed. EnzymeML serves as a seamless communication channel between experimental platforms, electronic lab notebooks, tools for modeling of enzyme kinetics, publication platforms and enzymatic reaction databases. EnzymeML is open and transparent, and invites the community to contribute. All documents and codes are freely available at https://enzymeml.org .

Considerations towards the better integration of epidemiology into quantitative risk assessment

Environmental epidemiology has proven critical to study various associations between environmental exposures and adverse human health effects. However, there is a perception that it often does not sufficiently inform quantitative risk assessment. To help address this concern, in 2017, the Health and Environmental Sciences Institute initiated a project engaging the epidemiology, exposure science, and risk assessment communities with tripartite representation from government agencies, industry, and academia, in a dialogue on the use of environmental epidemiology for quantitative risk assessment and public health decision making. As part of this project, four meetings attended by experts in epidemiology, exposure science, toxicology, statistics, and risk assessment, as well as one additional meeting engaging funding agencies, were organized to explore incentives and barriers to realizing the full potential of epidemiological data in quantitative risk assessment. A set of questions was shared with workshop participants prior to the meetings, and two case studies were used to support the discussion. Five key ideas emerged from these meetings as areas of desired improvement to ensure that human data can more consistently become an integral part of quantitative risk assessment: 1) reducing confirmation and publication bias, 2) increasing communication with funding agencies to raise awareness of research needs, 3) developing alternative funding channels targeted to support quantitative risk assessment, 4) making data available for reuse and analysis, and 5) developing cross-disciplinary and cross-sectoral interactions, collaborations, and training. We explored and integrated these themes into a roadmap illustrating the need for a multi-stakeholder effort to ensure that epidemiological data can fully contribute to the quantitative evaluation of human health risks, and to build confidence in a reliable decision-making process that leverages the totality of scientific evidence.

A meta-epidemiological assessment of transparency indicators of infectious disease models

Mathematical models have become very influential, especially during the COVID-19 pandemic. Data and code sharing are indispensable for reproducing them, protocol registration may be useful sometimes, and declarations of conflicts of interest (COIs) and of funding are quintessential for transparency. Here, we evaluated these features in publications of infectious disease-related models and assessed whether there were differences before and during the COVID-19 pandemic and for COVID-19 models versus models for other diseases. We analysed all PubMed Central open access publications of infectious disease models published in 2019 and 2021 using previously validated text mining algorithms of transparency indicators. We evaluated 1338 articles: 216 from 2019 and 1122 from 2021 (of which 818 were on COVID-19); almost a six-fold increase in publications within the field. 511 (39.2%) were compartmental models, 337 (25.2%) were time series, 279 (20.9%) were spatiotemporal, 186 (13.9%) were agent-based and 25 (1.9%) contained multiple model types. 288 (21.5%) articles shared code, 332 (24.8%) shared data, 6 (0.4%) were registered, and 1197 (89.5%) and 1109 (82.9%) contained COI and funding statements, respectively. There was no major changes in transparency indicators between 2019 and 2021. COVID-19 articles were less likely to have funding statements and more likely to share code. Further validation was performed by manual assessment of 10% of the articles identified by text mining as fulfilling transparency indicators and of 10% of the articles lacking them. Correcting estimates for validation performance, 26.0% of papers shared code and 41.1% shared data. On manual assessment, 5/6 articles identified as registered had indeed been registered. Of articles containing COI and funding statements, 95.8% disclosed no conflict and 11.7% reported no funding. Transparency in infectious disease modelling is relatively low, especially for data and code sharing. This is concerning, considering the nature of this research and the heightened influence it has acquired.

Un-biased housekeeping gene panel selection for high-validity gene expression analysis

Differential gene expression normalised to a single housekeeping (HK) is used to identify disease mechanisms and therapeutic targets. HK gene selection is often arbitrary, potentially introducing systematic error and discordant results. Here we examine these risks in a disease model of brain hypoxia. We first identified the eight most frequently used HK genes through a systematic review. However, we observe that in both ex-vivo and in vivo, their expression levels varied considerably between conditions. When applying these genes to normalise expression levels of the validated stroke target gene, inducible Nox4, we obtained opposing results. As an alternative tool for unbiased HK gene selection, software tools exist but are limited to individual datasets lacking genome-wide search capability and user-friendly interfaces. We, therefore, developed the HouseKeepR algorithm to rapidly analyse multiple gene expression datasets in a disease-specific manner and rank HK gene candidates according to stability in an unbiased manner. Using a panel of de novo top-ranked HK genes for brain hypoxia, but not single genes, Nox4 induction was consistently reproduced. Thus, differential gene expression analysis is best normalised against a HK gene panel selected in an unbiased manner. HouseKeepR is the first user-friendly, bias-free, and broadly applicable tool to automatically propose suitable HK genes in a tissue- and disease-dependent manner.

Open Science Practices in Gambling Research Publications (2016-2019): A Scoping Review

The replication crisis has stimulated researchers around the world to adopt open science research practices intended to reduce publication bias and improve research quality. Open science practices include study pre-registration, open data, open access, and avoiding methods that can lead to publication bias and low replication rates. Although gambling studies uses similar research methods as behavioral research fields that have struggled with replication, we know little about the uptake of open science research practices in gambling-focused research. We conducted a scoping review of 500 recent (1/1/2016–12/1/2019) studies focused on gambling and problem gambling to examine the use of open science and transparent research practices. Our results showed that a small percentage of studies used most practices: whereas 54.6% (95% CI: [50.2, 58.9]) of studies used at least one of nine open science practices, each practice’s prevalence was: 1.6% for pre-registration (95% CI: [0.8, 3.1]), 3.2% for open data (95% CI: [2.0, 5.1]), 0% for open notebook, 35.2% for open access (95% CI: [31.1, 39.5]), 7.8% for open materials (95% CI: [5.8, 10.5]), 1.4% for open code (95% CI: [0.7, 2.9]), and 15.0% for preprint posting (95% CI: [12.1, 18.4]). In all, 6.4% (95% CI: [4.6, 8.9]) of the studies included a power analysis and 2.4% (95% CI: [1.4, 4.2]) were replication studies. Exploratory analyses showed that studies that used any open science practice, and open access in particular, had higher citation counts. We suggest several practical ways to enhance the uptake of open science principles and practices both within gambling studies and in science more generally.

Considerations for an integrated population health databank in Africa: lessons from global best practices

Background: The rising digitisation and proliferation of data sources and repositories cannot be ignored. This trend expands opportunities to integrate and share population health data. Such platforms have many benefits, including the potential to efficiently translate information arising from such data to evidence needed to address complex global health challenges. There are pockets of quality data on the continent that may benefit from greater integration. Integration of data sources is however under-explored in Africa. The aim of this article is to identify the requirements and provide practical recommendations for developing a multi-consortia public and population health data-sharing framework for Africa.

Methods: We conducted a narrative review of global best practices and policies on data sharing and its optimisation. We searched eight databases for publications and undertook an iterative snowballing search of articles cited in the identified publications. The Leximancer software © enabled content analysis and selection of a sample of the most relevant articles for detailed review. Themes were developed through immersion in the extracts of selected articles using inductive thematic analysis. We also performed interviews with public and population health stakeholders in Africa to gather their experiences, perceptions, and expectations of data sharing.

Results: Our findings described global stakeholder experiences on research data sharing. We identified some challenges and measures to harness available resources and incentivise data sharing.  We further highlight progress made by the different groups in Africa and identified the infrastructural requirements and considerations when implementing data sharing platforms. Furthermore, the review suggests key reforms required, particularly in the areas of consenting, privacy protection, data ownership, governance, and data access.

Conclusions: The findings underscore the critical role of inclusion, social justice, public good, data security, accountability, legislation, reciprocity, and mutual respect in developing a responsive, ethical, durable, and integrated research data sharing ecosystem.

Transparent and Reproducible Research Practices in the Surgical Literature

INTRODUCTION

Previous studies have established a baseline of minimal reproducibility in the social science and biomedical literature. Clinical research is especially deficient in factors of reproducibility. Surgical journals contain fewer clinical trials than non-surgical areas of medicine, suggesting that it should be easier to reproduce the outcomes of surgical literature.

METHODS

In this study, we evaluated a broad range of indicators related to transparency and reproducibility in a random sample of 387 articles published in Surgery journals between 2014 and 2018.

RESULTS

A small minority of our sample made available their materials (5.3%, 95% C.I. 2.4%-8.2%), protocols (1.2%, 0-2.5%), data (2.5%, 0.7%-4.2%), or analysis scripts (0.04%). Four studies were adequately pre-registered. No studies were explicit replications of previous literature. Most studies (58%), declined to provide a funding statement, while conflicts of interest were declared in a small fraction (9.3%). Most have not been cited by systematic reviews (83%) or meta-analyses (87%), and most were only accessible to paying subscribers (59%).

CONCLUSIONS

The transparency of the surgical literature could improve with adherence to baseline standards of reproducibility.

Rigour and reproducibility in Canadian research: call for a coordinated approach

Shortcomings in the rigour and reproducibility of research have become well-known issues and persist despite repeated calls for improvement. A coordinated effort among researchers, institutions, funders, publishers, learned societies, and regulators may be the most effective way of tackling these issues. The UK Reproducibility Network (UKRN) has fostered collaboration across various stakeholders in research and are creating the infrastructure necessary to advance rigorous and reproducible research practices across the United Kingdom. Other Reproducibility Networks, modelled on UKRN, are now emerging in other countries. Canada could benefit from a comparable network to unify the voices around research quality and maximize the value of Canadian research.

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017)

Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.

Effects of Different Therapeutic Exercise Modalities on Migraine or Tension-Type Headache: A Systematic Review and Meta-Analysis with a Replicability Analysis

The primary aim of this study was to review the effect of exercise in comparison with a non-active treatment on pain intensity, frequency of headache episodes, headache duration, quality of life, medication use, and psychological symptoms, in patients with migraine or tension-type headache (TTH). A systematic search was conducted in various electronic databases to identify all relevant studies: Medline (PubMed), PEDro, EBSCO and Google Scholar. Clinical trials assessing the effects of exercise interventions in patients with primary headaches were selected. Methodological quality was evaluated using the Cochrane Risk of Bias Tool and PEDro scale and qualitative analysis was based on classifying the results into levels of evidence according to the GRADE. 19 studies (2776 participants; 85% female) were included. The meta-analysis showed statistically significant differences in pain intensity for aerobic training in patients with migraine (SMD = -0.65; 95% CI = -1.07 to -0.22, very low certainty evidence) and for strength training in patients with TTH (SMD = -0.84; 95% CI = -1.68 to- -0.01, very low certainty evidence). Statistically significant differences were also found in the medication use (SMD = -0.51; 95% CI = -0.85 to -0.17, low certainty evidence). Low transparency, replicability and high risk of bias were found. Aerobic training has a small to moderate clinical effect on pain intensity and medication use on migraine patients, with very low to low certainty of evidence. Strength training showed a moderate clinical effect with very low quality of evidence in patients with TTH. Exercise could be considered as clinically relevant for the management of patients with primary headaches, but the presence of low certainty of evidence and low transparency and replicability limited its clinical application. PERSPECTIVE: This article presents current evidence about exercise interventions in patients with primary headaches, including migraine and tension-type headache. Existing findings are reviewed, and relevant data are provided on the effectiveness of each exercise modality, as well as its certainty of evidence and clinical applicability.

Teaching reproducible research for medical students and postgraduate pharmaceutical scientists

In medicine and other academic settings, (doctoral) students often work in interdisciplinary teams together with researchers of pharmaceutical sciences, natural sciences in general, or biostatistics. They should be fundamentally taught good research practices, especially in terms of statistical analysis. This includes reproducibility as a central aspect. Acknowledging that even experienced researchers and supervisors might be unfamiliar with necessary aspects of a perfectly reproducible workflow, a lecture series on reproducible research (RR) was developed for young scientists in clinical pharmacology. The pilot series highlighted definitions of RR, reasons for RR, potential merits of RR, and ways to work accordingly. In trying to actually reproduce a published analysis, several practical obstacles arose. In this article, reproduction of a working example is commented to emphasize the manifold facets of RR, to provide possible explanations for difficulties and solutions, and to argue that harmonized curricula for (quantitative) clinical researchers should include RR principles. These experiences should raise awareness among educators and students, supervisors and young scientists. RR working habits are not only beneficial for ourselves or our students, but also for other researchers within an institution, for scientific partners, for the scientific community, and eventually for the public profiting from research findings.

Who funded the research behind the Oxford-AstraZeneca COVID-19 vaccine?

OBJECTIVES

The Oxford-AstraZeneca COVID-19 vaccine (ChAdOx1 nCoV-19, Vaxzevira or Covishield) builds on two decades of research and development (R&D) into chimpanzee adenovirus-vectored vaccine (ChAdOx) technology at the University of Oxford. This study aimed to approximate the funding for the R&D of ChAdOx and the Oxford-AstraZeneca vaccine and to assess the transparency of funding reporting mechanisms.

METHODS

We conducted a scoping review and publication history analysis of the principal investigators to reconstruct R&D funding the ChAdOx technology. We matched award numbers with publicly accessible grant databases. We filed freedom of information (FOI) requests to the University of Oxford for the disclosure of all grants for ChAdOx R&D.

RESULTS

We identified 100 peer-reviewed articles relevant to ChAdOx technology published between January 2002 and October 2020, extracting 577 mentions of funding bodies from acknowledgements. Government funders from overseas (including the European Union) were mentioned 158 times (27.4%), the UK government 147 (25.5%) and charitable funders 138 (23.9%). Grant award numbers were identified for 215 (37.3%) mentions; amounts were publicly available for 121 (21.0%). Based on the FOIs, until December 2019, the biggest funders of ChAdOx R&D were the European Commission (34.0%), Wellcome Trust (20.4%) and Coalition for Epidemic Preparedness Innovations (17.5%). Since January 2020, the UK government contributed 95.5% of funding identified. The total identified R&D funding was £104 226 076 reported in the FOIs and £228 466 771 reconstructed from the literature search.

CONCLUSION

Our study approximates that public and charitable financing accounted for 97%-99% of identifiable funding for the ChAdOx vaccine technology research at the University of Oxford underlying the Oxford-AstraZeneca vaccine until autumn 2020. We encountered a lack of transparency in research funding reporting.

Signaling Targets Related to Antiobesity Effects of Capsaicin: A Scoping Review

The search for new antiobesogenic agents is increasing because of the current obesity pandemic. Capsaicin (Caps), an exogenous agonist of the vanilloid receptor of transient potential type 1 (TRPV1), has shown promising results in the treatment of obesity. This scoping review aims to verify the pathways mediating the effects of Caps in obesity and the different methods adopted to identify these pathways. The search was carried out using data from the EMBASE, MEDLINE (PubMed), Web of Science, and SCOPUS databases. Studies considered eligible evaluated the mechanisms of action of Caps in obesity models or cell types involved in obesity. Nine studies were included and 100% (n = 6) of the in vivo studies showed a high risk of bias. Of the 9 studies, 66.6% (n = 6) administered Caps orally in the diet and 55.5% (n = 5) used a concentration of Caps of 0.01% in the diet. In vitro, the most tested concentration was 1 μM (88.9%; n = 8). Capsazepine was the antagonist chosen by 66.6% (n = 6) of the studies. Seven studies (77.8%) linked the antiobesogenic effects of Caps to TRPV1 activation and 3 (33.3%) indicated peroxisome proliferator-activated receptor (PPAR) involvement as an upstream connection to TRPV1, rather than a direct metabolic target of Caps. The main secondary effects of Caps were lower weight gain (33.3%; n = 3) or loss (22.2%; n = 2), greater improvement in lipid profile (33.3%; n = 3), lower white adipocyte adipogenesis (33.3%; n = 3), browning process activation (44.4%; n = 4), and higher brown adipocyte activity (33.3%; n = 3) compared with those of the control treatment. Some studies have shown that PPAR agonists modulate TRPV1 activity, and no study has evaluated the simultaneous antagonism of these 2 receptors. Consequently, further studies are necessary to elucidate the role of each of these signaling molecules in the antiobesogenic effects of Caps.

Reproducibility of Research During COVID-19: Examining the Case of Population Density and the Basic Reproductive Rate from the Perspective of Spatial Analysis

The emergence of the novel SARS-CoV-2 coronavirus and the global COVID-19 pandemic in 2019 led to explosive growth in scientific research. Alas, much of the research in the literature lacks conditions to be reproducible, and recent publications on the association between population density and the basic reproductive number of SARS-CoV-2 are no exception. Relatively few papers share code and data sufficiently, which hinders not only verification but additional experimentation. In this article, an example of reproducible research shows the potential of spatial analysis for epidemiology research during COVID-19. Transparency and openness means that independent researchers can, with only modest efforts, verify findings and use different approaches as appropriate. Given the high stakes of the situation, it is essential that scientific findings, on which good policy depends, are as robust as possible; as the empirical example shows, reproducibility is one of the keys to ensure this.

Raising the bar: improving methodological rigour in cognitive alcohol research

BACKGROUND AND AIMS

A range of experimental paradigms claim to measure the cognitive processes underpinning alcohol use, suggesting that heightened attentional bias, greater approach tendencies and reduced cue-specific inhibitory control are important drivers of consumption. This paper identifies methodological shortcomings within this broad domain of research and exemplifies them in studies focused specifically on alcohol-related attentional bias.

ARGUMENT AND ANALYSIS

We highlight five main methodological issues: (i) the use of inappropriately matched control stimuli; (ii) opacity of stimulus selection and validation procedures; (iii) a credence in noisy measures; (iv) a reliance on unreliable tasks; and (v) variability in design and analysis. This is evidenced through a review of alcohol-related attentional bias (64 empirical articles, 68 tasks), which reveals the following: only 53% of tasks use appropriately matched control stimuli; as few as 38% report their stimulus selection and 19% their validation procedures; less than 28% used indices capable of disambiguating attentional processes; 22% assess reliability; and under 2% of studies were pre-registered.

CONCLUSIONS

Well-matched and validated experimental stimuli, the development of reliable cognitive tasks and explicit assessment of their psychometric properties, and careful consideration of behavioural indices and their analysis will improve the methodological rigour of cognitive alcohol research. Open science principles can facilitate replication and reproducibility in alcohol research.

Rates and predictors of data and code sharing in the medical and health sciences: Protocol for a systematic review and individual participant data meta-analysis

Numerous studies have demonstrated low but increasing rates of data and code sharing within medical and health research disciplines. However, it remains unclear how commonly data and code are shared across all fields of medical and health research, as well as whether sharing rates are positively associated with implementation of progressive policies by publishers and funders, or growing expectations from the medical and health research community at large. Therefore this systematic review aims to synthesise the findings of medical and health science studies that have empirically investigated the prevalence of data or code sharing, or both. Objectives include the investigation of: (i) the prevalence of public sharing of research data and code alongside published articles (including preprints), (ii) the prevalence of private sharing of research data and code in response to reasonable requests, and (iii) factors associated with the sharing of either research output (e.g., the year published, the publisher's policy on sharing, the presence of a data or code availability statement). It is hoped that the results will provide some insight into how often research data and code are shared publicly and privately, how this has changed over time, and how effective some measures such as the institution of data sharing policies and data availability statements have been in motivating researchers to share their underlying data and code.

Applied machine learning in cancer research: A systematic review for patient diagnosis, classification and prognosis

Artificial Intelligence (AI) has recently altered the landscape of cancer research and medical oncology using traditional Machine Learning (ML) algorithms and cutting-edge Deep Learning (DL) architectures. In this review article we focus on the ML aspect of AI applications in cancer research and present the most indicative studies with respect to the ML algorithms and data used. The PubMed and dblp databases were considered to obtain the most relevant research works of the last five years. Based on a comparison of the proposed studies and their research clinical outcomes concerning the medical ML application in cancer research, three main clinical scenarios were identified. We give an overview of the well-known DL and Reinforcement Learning (RL) methodologies, as well as their application in clinical practice, and we briefly discuss Systems Biology in cancer research. We also provide a thorough examination of the clinical scenarios with respect to disease diagnosis, patient classification and cancer prognosis and survival. The most relevant studies identified in the preceding year are presented along with their primary findings. Furthermore, we examine the effective implementation and the main points that need to be addressed in the direction of robustness, explainability and transparency of predictive models. Finally, we summarize the most recent advances in the field of AI/ML applications in cancer research and medical oncology, as well as some of the challenges and open issues that need to be addressed before data-driven models can be implemented in healthcare systems to assist physicians in their daily practice.

Randomization, blinding, data handling and sample size estimation in papers published in Veterinary Anaesthesia and Analgesia in 2009 and 2019

OBJECTIVE

To evaluate reporting of items indicative of bias and weak study design.

STUDY DESIGN

Literature survey.

POPULATION

Papers published in Veterinary Anaesthesia and Analgesia.

METHODS

Reporting of randomization, blinding, sample size estimation and data exclusion were compared for papers published separated by a 10 year interval. A reporting rate of more than 95% was considered ideal. The availability of data supporting results in a publicly accessible repository was also assessed. Selected papers were randomized and identifiers removed for review, with data from 59 (57 in 2009, two in 2008) and 56 (52 in 2019, four in 2018) papers analyzed. Items were categorized for completeness of reporting using a previously published operationalized checklist. Two reviewers reviewed all papers independently.

RESULTS

Full reporting of randomization increased over time from 13.6% to 85.7% [95% confidence interval (CI), 57.8-86.6%; p < 0.0001], as did sample size estimation (from 0% to 20%; 95% CI, 7.6-32.4%; p = 0.002). Reporting of blinding (49.2% and 50.0%; 95% CI, -18.3% to 20.0%; p = 1.0) and exclusions of samples/animals (39.0% and 50.0%; 95% CI, -8.8% to 30.8%; p = 0.3) did not change significantly. Data availability was low (2008/2009, zero papers; 2018/2019, two papers). None of the items studied exceeded the predetermined ideal reporting rate.

CONCLUSIONS AND CLINICAL RELEVANCE

These results indicate that reporting quality remains low, with a risk of bias.

NPARS-A Novel Approach to Address Accuracy and Reproducibility in Genomic Data Science

Background: Accuracy and reproducibility are vital in science and presents a significant challenge in the emerging discipline of data science, especially when the data are scientifically complex and massive in size. Further complicating matters, in the field of genomic-based science high-throughput sequencing technologies generate considerable amounts of data that needs to be stored, manipulated, and analyzed using a plethora of software tools. Researchers are rarely able to reproduce published genomic studies. Results: Presented is a novel approach which facilitates accuracy and reproducibility for large genomic research data sets. All data needed is loaded into a portable local database, which serves as an interface for well-known software frameworks. These include python-based Jupyter Notebooks and the use of RStudio projects and R markdown. All software is encapsulated using Docker containers and managed by Git, simplifying software configuration management. Conclusion: Accuracy and reproducibility in science is of a paramount importance. For the biomedical sciences, advances in high throughput technologies, molecular biology and quantitative methods are providing unprecedented insights into disease mechanisms. With these insights come the associated challenge of scientific data that is complex and massive in size. This makes collaboration, verification, validation, and reproducibility of findings difficult. To address these challenges the NGS post-pipeline accuracy and reproducibility system (NPARS) was developed. NPARS is a robust software infrastructure and methodology that can encapsulate data, code, and reporting for large genomic studies. This paper demonstrates the successful use of NPARS on large and complex genomic data sets across different computational platforms.

Data sharing practices of health researchers in Africa: a scoping review protocol

OBJECTIVE

The aim of the review is to map the existing evidence regarding the data-sharing practices of health researchers in African countries. This review will also identify perceptions; barriers; facilitators; ethical-, legal-, and author-reported recommendations; institutional- and funding-related aspects that are being considered by African health researchers on data sharing in Africa and, as a result, identify areas for development and improvement in health care on the continent.

INTRODUCTION

The sharing of health-related data has been widely discussed in the literature. However, sharing health-related data has yet to become a common practice among health researchers in Africa, which bears a large burden of the global health diseases. The sharing of health research data could lead to greater development and improvement in health care in Africa.

INCLUSION CRITERIA

This review will incorporate studies that report on data sharing among health researchers in Africa. All primary, secondary, and gray literature that report on the practice of data sharing among health researchers in Africa will be included. Studies on data sharing on topics other than health-related data will be excluded. No language restrictions will be applied.

METHODS

The JBI scoping review methodological framework will be adopted. An initial search of databases such as MEDLINE (PubMed), Scopus, LILAC, and Web of Science will be conducted. All search results will be screened and relevant data extracted by two independent reviewers. Data extracted will be exported into JBI SUMARI. The findings will be presented in the final scoping review report and illustrated in a PRISMA flow diagram.

Measuring and interpreting pervasive heterogeneity, poikilosis

Measurements are widely used in science, engineering, industry, and trade. They form the basis for experimental scientific research, approach, and progress; however, their foundations are seldom thought or questioned. Recently poikilosis, pervasive heterogeneity ranging from subatomic level to biosphere, was introduced. Poikilosis makes single point measurements and estimates obsolete and irrelevant as measurands display intervals of magnitudes. Consideration of poikilosis requires new lines of thinking in experimental design, conduction of studies, data analysis and interpretation. Measurements of poikilosis must consider lagom, normal, variation extent. Measurements, measures, and measurands as well as the measuring systems and uncertainties are discussed from the perspective of poikilosis. New systematics is introduced for description of uncertainty in measurements and for types of experimental designs. Poikilosis-aware experimenting, data analysis and interpretation are discussed. Instructions are provided for how to measure lagom and non-lagom effects of poikilosis. Consideration of poikilosis can solve scientific controversies and enigmas and can allow novel insight into systems, processes, mechanisms, and reactions and their interpretation, understanding, and manipulation. Furthermore, it will increase reproducibility of measurements and studies.

A meta-epidemiological study found lack of transparency and poor reporting of disproportionality analyses for signal detection in pharmacovigilance databases

OBJECTIVE

To review and appraise methods and reporting characteristics of pharmacovigilance disproportionality analyses.

STUDY DESIGN AND SETTING

We randomly selected 100 disproportionality analyses indexed in Medline found during a systematic literature search. We then extracted and synthetized methodological and reporting characteristics using 7 key items: 1) title transparency; 2) protocol pre-registration; 3) date of data extraction and analysis; 4) outcome, population, exposure and comparator definitions; 5) adjustment and stratification of results; 6) method and threshold for signal detection; 7) secondary and sensitivity analyses.

RESULTS

We found that methods used to generate disproportionality signals were extremely heterogeneous; there were nearly as many unique analyses as studies. The authors used various populations, methods, signal detection thresholds, adjustment or stratification variables, generally without justification for their choice or pre-specification in protocols. Moreover, 78% of studies failed to report methods for case, adverse drug reactions or comparator selection and 32 studies did not define the threshold for signal generation.

CONCLUSION

Our survey raises major concerns regarding all aspects of disproportionality analyses that could lead to misleading results and generate unjustified alarms. We advocate for a strong and transparent rationale for variable selection, choice of population and comparators pre-specified in a protocol and assessed by sensitivity analyses.

Evaluating Reproducibility and Transparency in Emergency Medicine Publications

INTRODUCTION

We aimed to assess the reproducibility of empirical research by determining the availability of components required for replication of a study, including materials, raw data, analysis scripts, protocols, and preregistration.

METHODS

We used the National Library of Medicine catalog to identify MEDLINE-indexed emergency medicine (EM) journals. Thirty journals met the inclusion criteria. From January 1, 2014-December 31, 2018, 300 publications were randomly sampled using a PubMed search. Additionally, we included four high-impact general medicine journals, which added 106 publications. Two investigators were blinded for independent extraction. Extracted data included statements regarding the availability of materials, data, analysis scripts, protocols, and registration.

RESULTS

After the search, we found 25,473 articles, from which we randomly selected 300. Of the 300, only 287 articles met the inclusion criteria. Additionally, we added 106 publications from high-impact journals of which 77 met the inclusion criteria. Together, 364 publications were included, of which 212 articles contained empirical data to analyze. Of the eligible empirical articles, 2.49%, (95% confidence interval [CI], 0.33% to 4.64%] provided a material statement, 9.91% (95% CI, 5.88% to 13.93%) provided a data statement, 0 provided access to analysis scripts, 25.94% (95% CI, 20.04% to 31.84%) linked the protocol, and 39.15% (95% CI, 32.58% to 45.72%) were preregistered.

CONCLUSION

Studies in EM lack indicators required for reproducibility. The majority of studies fail to report factors needed to reproduce research to ensure credibility. Thus, an intervention is required and can be achieved through the collaboration of researchers, peer reviewers, funding agencies, and journals.

Knowledge and Attitudes Among Life Scientists Toward Reproducibility Within Journal Articles: A Research Survey

We constructed a survey to understand how authors and scientists view the issues around reproducibility, focusing on interactive elements such as interactive figures embedded within online publications, as a solution for enabling the reproducibility of experiments. We report the views of 251 researchers, comprising authors who have published in eLIFE Sciences, and those who work at the Norwich Biosciences Institutes (NBI). The survey also outlines to what extent researchers are occupied with reproducing experiments themselves. Currently, there is an increasing range of tools that attempt to address the production of reproducible research by making code, data, and analyses available to the community for reuse. We wanted to collect information about attitudes around the consumer end of the spectrum, where life scientists interact with research outputs to interpret scientific results. Static plots and figures within articles are a central part of this interpretation, and therefore we asked respondents to consider various features for an interactive figure within a research article that would allow them to better understand and reproduce a published analysis. The majority (91%) of respondents reported that when authors describe their research methodology (methods and analyses) in detail, published research can become more reproducible. The respondents believe that having interactive figures in published papers is a beneficial element to themselves, the papers they read as well as to their readers. Whilst interactive figures are one potential solution for consuming the results of research more effectively to enable reproducibility, we also review the equally pressing technical and cultural demands on researchers that need to be addressed to achieve greater success in reproducibility in the life sciences.

A meta-review of transparency and reproducibility-related reporting practices in published meta-analyses on clinical psychological interventions (2000-2020)

Meta-analysis is a powerful and important tool to synthesize the literature about a research topic. Like other kinds of research, meta-analyses must be reproducible to be compliant with the principles of the scientific method. Furthermore, reproducible meta-analyses can be easily updated with new data and reanalysed applying new and more refined analysis techniques. We attempted to empirically assess the prevalence of transparency and reproducibility-related reporting practices in published meta-analyses from clinical psychology by examining a random sample of 100 meta-analyses. Our purpose was to identify the key points that could be improved, with the aim of providing some recommendations for carrying out reproducible meta-analyses. We conducted a meta-review of meta-analyses of psychological interventions published between 2000 and 2020. We searched PubMed, PsycInfo and Web of Science databases. A structured coding form to assess transparency indicators was created based on previous studies and existing meta-analysis guidelines. We found major issues concerning: completely reproducible search procedures report, specification of the exact method to compute effect sizes, choice of weighting factors and estimators, lack of availability of the raw statistics used to compute the effect size and of interoperability of available data, and practically total absence of analysis script code sharing. Based on our findings, we conclude with recommendations intended to improve the transparency, openness, and reproducibility-related reporting practices of meta-analyses in clinical psychology and related areas.

Reproducible validation and replication studies in nanoscale physics

Credibility building activities in computational research include verification and validation, reproducibility and replication, and uncertainty quantification. Though orthogonal to each other, they are related. This paper presents validation and replication studies in electromagnetic excitations on nanoscale structures, where the quantity of interest is the wavelength at which resonance peaks occur. The study uses the open-source software PyGBe: a boundary element solver with treecode acceleration and GPU capability. We replicate a result by Rockstuhl et al. (2005, doi:10/dsxw9d) with a two-dimensional boundary element method on silicon carbide (SiC) particles, despite differences in our method. The second replication case from Ellis et al. (2016, doi:10/f83zcb) looks at aspect ratio effects on high-order modes of localized surface phonon-polariton nanostructures. The results partially replicate: the wavenumber position of some modes match, but for other modes they differ. With virtually no information about the original simulations, explaining the discrepancies is not possible. A comparison with experiments that measured polarized reflectance of SiC nano pillars provides a validation case. The wavenumber of the dominant mode and two more do match, but differences remain in other minor modes. Results in this paper were produced with strict reproducibility practices, and we share reproducibility packages for all, including input files, execution scripts, secondary data, post-processing code and plotting scripts, and the figures (deposited in Zenodo). In view of the many challenges faced, we propose that reproducible practices make replication and validation more feasible. This article is part of the theme issue 'Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico'.