Recommendations to enhance rigor and reproducibility in biomedical research

Best practices for data management and sharing in experimental biomedical research

Effective data management is crucial for scientific integrity and reproducibility, a cornerstone of scientific progress. Well-organized and well-documented data enable validation and building on results. Data management encompasses activities including organization, documentation, storage, sharing, and preservation. Robust data management establishes credibility, fostering trust within the scientific community and benefiting researchers' careers. In experimental biomedicine, comprehensive data management is vital due to the typically intricate protocols, extensive metadata, and large datasets. Low-throughput experiments, in particular, require careful management to address variations and errors in protocols and raw data quality. Transparent and accountable research practices rely on accurate documentation of procedures, data collection, and analysis methods. Proper data management ensures long-term preservation and accessibility of valuable datasets. Well-managed data can be revisited, contributing to cumulative knowledge and potential new discoveries. Publicly funded research has an added responsibility for transparency, resource allocation, and avoiding redundancy. Meeting funding agency expectations increasingly requires rigorous methodologies, adherence to standards, comprehensive documentation, and widespread sharing of data, code, and other auxiliary resources. This review provides critical insights into raw and processed data, metadata, high-throughput versus low-throughput datasets, a common language for documentation, experimental and reporting guidelines, efficient data management systems, sharing practices, and relevant repositories. We systematically present available resources and optimal practices for wide use by experimental biomedical researchers.

scDAPP: a comprehensive single-cell transcriptomics analysis pipeline optimized for cross-group comparison

Single-cell transcriptomics profiling has increasingly been used to evaluate cross-group differences in cell population and cell-type gene expression. This often leads to large datasets with complex experimental designs that need advanced comparative analysis. Concurrently, bioinformatics software and analytic approaches also become more diverse and constantly undergo improvement. Thus, there is an increased need for automated and standardized data processing and analysis pipelines, which should be efficient and flexible too. To address these, we develop the single-cell Differential Analysis and Processing Pipeline (scDAPP), a R-based workflow for comparative analysis of single cell (or nucleus) transcriptomic data between two or more groups and at the levels of single cells or "pseudobulking" samples. The pipeline automates many steps of pre-processing using data-learnt parameters, uses previously benchmarked software, and generates comprehensive intermediate data and final results that are valuable for both beginners and experts of scRNA-seq analysis. Moreover, the analytic reports, augmented by extensive data visualization, increase the transparency of computational analysis and parameter choices, while facilitate users to go seamlessly from raw data to biological interpretation. Availability and Implementation: scDAPP is freely available for non-commercial usage as an R package under the MIT license. Source code, documentation and sample data are available at the GitHub (https://github.com/bioinfoDZ/scDAPP).

Packaging and containerization of computational methods

Methods for analyzing the full complement of a biomolecule type, e.g., proteomics or metabolomics, generate large amounts of complex data. The software tools used to analyze omics data have reshaped the landscape of modern biology and become an essential component of biomedical research. These tools are themselves quite complex and often require the installation of other supporting software, libraries and/or databases. A researcher may also be using multiple different tools that require different versions of the same supporting materials. The increasing dependence of biomedical scientists on these powerful tools creates a need for easier installation and greater usability. Packaging and containerization are different approaches to satisfy this need by delivering omics tools already wrapped in additional software that makes the tools easier to install and use. In this systematic review, we describe and compare the features of prominent packaging and containerization platforms. We outline the challenges, advantages and limitations of each approach and some of the most widely used platforms from the perspectives of users, software developers and system administrators. We also propose principles to make the distribution of omics software more sustainable and robust to increase the reproducibility of biomedical and life science research.

CREDO: a friendly Customizable, REproducible, DOcker file generator for bioinformatics applications

BACKGROUND

The analysis of large and complex biological datasets in bioinformatics poses a significant challenge to achieving reproducible research outcomes due to inconsistencies and the lack of standardization in the analysis process. These issues can lead to discrepancies in results, undermining the credibility and impact of bioinformatics research and creating mistrust in the scientific process. To address these challenges, open science practices such as sharing data, code, and methods have been encouraged.

RESULTS

CREDO, a Customizable, REproducible, DOcker file generator for bioinformatics applications, has been developed as a tool to moderate reproducibility issues by building and distributing docker containers with embedded bioinformatics tools. CREDO simplifies the process of generating Docker images, facilitating reproducibility and efficient research in bioinformatics. The crucial step in generating a Docker image is creating the Dockerfile, which requires incorporating heterogeneous packages and environments such as Bioconductor and Conda. CREDO stores all required package information and dependencies in a Github-compatible format to enhance Docker image reproducibility, allowing easy image creation from scratch. The user-friendly GUI and CREDO's ability to generate modular Docker images make it an ideal tool for life scientists to efficiently create Docker images. Overall, CREDO is a valuable tool for addressing reproducibility issues in bioinformatics research and promoting open science practices.

CHOIR improves significance-based detection of cell types and states from single-cell data

Clustering is a critical step in the analysis of single-cell data, as it enables the discovery and characterization of putative cell types and states. However, most popular clustering tools do not subject clustering results to statistical inference testing, leading to risks of overclustering or underclustering data and often resulting in ineffective identification of cell types with widely differing prevalence. To address these challenges, we present CHOIR (clustering hierarchy optimization by iterative random forests), which applies a framework of random forest classifiers and permutation tests across a hierarchical clustering tree to statistically determine which clusters represent distinct populations. We demonstrate the enhanced performance of CHOIR through extensive benchmarking against 14 existing clustering methods across 100 simulated and 4 real single-cell RNA-seq, ATAC-seq, spatial transcriptomic, and multi-omic datasets. CHOIR can be applied to any single-cell data type and provides a flexible, scalable, and robust solution to the important challenge of identifying biologically relevant cell groupings within heterogeneous single-cell data.

Computational reproducibility of Jupyter notebooks from biomedical publications

BACKGROUND

Jupyter notebooks facilitate the bundling of executable code with its documentation and output in one interactive environment, and they represent a popular mechanism to document and share computational workflows, including for research publications. The reproducibility of computational aspects of research is a key component of scientific reproducibility but has not yet been assessed at scale for Jupyter notebooks associated with biomedical publications.

APPROACH

We address computational reproducibility at 2 levels: (i) using fully automated workflows, we analyzed the computational reproducibility of Jupyter notebooks associated with publications indexed in the biomedical literature repository PubMed Central. We identified such notebooks by mining the article's full text, trying to locate them on GitHub, and attempting to rerun them in an environment as close to the original as possible. We documented reproduction success and exceptions and explored relationships between notebook reproducibility and variables related to the notebooks or publications. (ii) This study represents a reproducibility attempt in and of itself, using essentially the same methodology twice on PubMed Central over the course of 2 years, during which the corpus of Jupyter notebooks from articles indexed in PubMed Central has grown in a highly dynamic fashion.

RESULTS

Out of 27,271 Jupyter notebooks from 2,660 GitHub repositories associated with 3,467 publications, 22,578 notebooks were written in Python, including 15,817 that had their dependencies declared in standard requirement files and that we attempted to rerun automatically. For 10,388 of these, all declared dependencies could be installed successfully, and we reran them to assess reproducibility. Of these, 1,203 notebooks ran through without any errors, including 879 that produced results identical to those reported in the original notebook and 324 for which our results differed from the originally reported ones. Running the other notebooks resulted in exceptions.

CONCLUSIONS

We zoom in on common problems and practices, highlight trends, and discuss potential improvements to Jupyter-related workflows associated with biomedical publications.

Data-Sharing Across Otolaryngology: Comparing Journal Policies and Their Adherence to the FAIR Principles

OBJECTIVE

Data-sharing plays an essential role in advancing scientific understanding. Here, we aim to identify the commonalities and differences in data-sharing policies endorsed by otolaryngology journals and to assess their adherence to the FAIR (findable, accessible, interoperable, reusable) principles.

METHODS

Data-sharing policies were searched for among 111 otolaryngology journals, as listed by Scimago Journal & Country Rank. Policy extraction of the top biomedical journals as ranked by Google Scholar metrics were used as a comparison. The FAIR principles for scientific data management and stewardship were used for the extraction framework. This occurred in a blind, masked, and independent fashion.

RESULTS

Of the 111 ranked otolaryngology journals, 100 met inclusion criteria. Of those 100 journals, 79 provided data-sharing policies. There was a clear lack of standardization across policies, along with specific gaps in accessibility and reusability which need to be addressed. Seventy-two policies (of 79; 91%) designated that metadata should have globally unique and persistent identifiers. Seventy-one (of 79; 90%) policies specified that metadata should clearly include the identifier of the data they describe. Fifty-six policies (of 79; 71%) outlined that metadata should be richly described with a plurality of accurate and relevant attributes.

CONCLUSION

Otolaryngology journals have varying data-sharing policies, and adherence to the FAIR principles appears to be moderate. This calls for increased data transparency, allowing for results to be reproduced, confirmed, and debated.

Hypermedia-based software architecture enables Test-Driven Development

Objectives

Using agile software development practices, develop and evaluate an architecture and implementation for reliable and user-friendly self-service management of bioinformatic data stored in the cloud.

Materials and methods

Comprehensive Oncology Research Environment (CORE) Browser is a new open-source web application for cancer researchers to manage sequencing data organized in a flexible format in Amazon Simple Storage Service (S3) buckets. It has a microservices- and hypermedia-based architecture, which we integrated with Test-Driven Development (TDD), the iterative writing of computable specifications for how software should work prior to development. Relying on repeating patterns found in hypermedia-based architectures, we hypothesized that hypermedia would permit developing test "templates" that can be parameterized and executed for each microservice, maximizing code coverage while minimizing effort.

Results

After one-and-a-half years of development, the CORE Browser backend had 121 test templates and 875 custom tests that were parameterized and executed 3031 times, providing 78% code coverage.

Discussion

Architecting to permit test reuse through a hypermedia approach was a key success factor for our testing efforts. CORE Browser's application of hypermedia and TDD illustrates one way to integrate software engineering methods into data-intensive networked applications. Separating bioinformatic data management from analysis distinguishes this platform from others in bioinformatics and may provide stable data management while permitting analysis methods to advance more rapidly.

Conclusion

Software engineering practices are underutilized in informatics. Similar informatics projects will more likely succeed through application of good architecture and automated testing. Our approach is broadly applicable to data management tools involving cloud data storage.

The Quartet Data Portal: integration of community-wide resources for multiomics quality control

The Quartet Data Portal facilitates community access to well-characterized reference materials, reference datasets, and related resources established based on a family of four individuals with identical twins from the Quartet Project. Users can request DNA, RNA, protein, and metabolite reference materials, as well as datasets generated across omics, platforms, labs, protocols, and batches. Reproducible analysis tools allow for objective performance assessment of user-submitted data, while interactive visualization tools support rapid exploration of reference datasets. A closed-loop "distribution-collection-evaluation-integration" workflow enables updates and integration of community-contributed multiomics data. Ultimately, this portal helps promote the advancement of reference datasets and multiomics quality control.

Integrative approaches to methods training for early-career scientists: Rationale and process evaluation of the first cohort of the Michigan Integrative Well-Being and Inequality Training Program

Background

The Michigan Integrative Well-Being and Inequality (MIWI) Training Program aims to provide state-of-the-art, interdisciplinary training to enhance the methodological skills of early-career scientists interested in integrative approaches to understanding health disparities. The goals of this paper are to describe the scientific rationale and core design elements of MIWI, and to conduct a process evaluation of the first cohort of trainees (called "scholars") to complete this program.

Methods

Mixed methods process evaluation of program components and assessment of trainee skills and network development of the first cohort (n = 15 scholars).

Results

The program drew 57 applicants from a wide range of disciplines. Of the 15 scholars in the first cohort, 53% (n = 8) identified as an underrepresented minority, 60% (n = 9) were within 2 years of completing their terminal degree, and most (n = 11, 73%) were from a social/behavioral science discipline (e.g., social work, public health). In the post-program evaluation, scholars rated their improvement in a variety of skills on a one (not at all) to five (greatly improved) scale. Areas of greatest growth included being an interdisciplinary researcher (mean = 4.47), developing new research collaborations (mean = 4.53), and designing a research study related to integrative health (mean = 4.27). The qualitative process evaluation indicated that scholars reported a strong sense of community and that the program broadened their research networks.

Conclusions

These findings have implications for National Institutes of Health (NIH) efforts to train early-career scientists, particularly from underrepresented groups, working at the intersection of multiple disciplines and efforts to support the formation of research networks.

Synaptonemal & CO analyzer: A tool for synaptonemal complex and crossover analysis in immunofluorescence images

During the formation of ova and sperm, homologous chromosomes get physically attached through the synaptonemal complex and exchange DNA at crossover sites by a process known as meiotic recombination. Chromosomes that do not recombine or have anomalous crossover distributions often separate poorly during the subsequent cell division and end up in abnormal numbers in ova or sperm, which can lead to miscarriage or developmental defects. Crossover numbers and distribution along the synaptonemal complex can be visualized by immunofluorescent microscopy. However, manual analysis of large numbers of cells is very time-consuming and a major bottleneck for recombination studies. Some image analysis tools have been created to overcome this situation, but they are not readily available, do not provide synaptonemal complex data, or do not tackle common experimental difficulties, such as overlapping chromosomes. To overcome these limitations, we have created and validated an open-source ImageJ macro routine that facilitates and speeds up the crossover and synaptonemal complex analyses in mouse chromosome spreads, as well as in other vertebrate species. It is free, easy to use and fulfills the recommendations for enhancing rigor and reproducibility in biomedical studies.

A Synthesis of the Formats for Correcting Erroneous and Fraudulent Academic Literature, and Associated Challenges

Academic publishing is undergoing a highly transformative process, and many established rules and value systems that are in place, such as traditional peer review (TPR) and preprints, are facing unprecedented challenges, including as a result of post-publication peer review. The integrity and validity of the academic literature continue to rely naively on blind trust, while TPR and preprints continue to fail to effectively screen out errors, fraud, and misconduct. Imperfect TPR invariably results in imperfect papers that have passed through varying levels of rigor of screening and validation. If errors or misconduct were not detected during TPR's editorial screening, but are detected at the post-publication stage, an opportunity is created to correct the academic record. Currently, the most common forms of correcting the academic literature are errata, corrigenda, expressions of concern, and retractions or withdrawals. Some additional measures to correct the literature have emerged, including manuscript versioning, amendments, partial retractions and retract and replace. Preprints can also be corrected if their version is updated. This paper discusses the risks, benefits and limitations of these forms of correcting the academic literature.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s10838-022-09607-4.

CIAlign: A highly customisable command line tool to clean, interpret and visualise multiple sequence alignments

Background

Throughout biology, multiple sequence alignments (MSAs) form the basis of much investigation into biological features and relationships. These alignments are at the heart of many bioinformatics analyses. However, sequences in MSAs are often incomplete or very divergent, which can lead to poor alignment and large gaps. This slows down computation and can impact conclusions without being biologically relevant. Cleaning the alignment by removing common issues such as gaps, divergent sequences, large insertions and deletions and poorly aligned sequence ends can substantially improve analyses. Manual editing of MSAs is very widespread but is time-consuming and difficult to reproduce.

Results

We present a comprehensive, user-friendly MSA trimming tool with multiple visualisation options. Our highly customisable command line tool aims to give intervention power to the user by offering various options, and outputs graphical representations of the alignment before and after processing to give the user a clear overview of what has been removed. The main functionalities of the tool include removing regions of low coverage due to insertions, removing gaps, cropping poorly aligned sequence ends and removing sequences that are too divergent or too short. The thresholds for each function can be specified by the user and parameters can be adjusted to each individual MSA. CIAlign is designed with an emphasis on solving specific and common alignment problems and on providing transparency to the user.

Conclusion

CIAlign effectively removes problematic regions and sequences from MSAs and provides novel visualisation options. This tool can be used to fine-tune alignments for further analysis and processing. The tool is aimed at anyone who wishes to automatically clean up parts of an MSA and those requiring a new, accessible way of visualising large MSAs.

BioVisReport: A Markdown-based lightweight website builder for reproducible and interactive visualization of results from peer-reviewed publications

Interactive visualization is an effective way to promote the reproducibility of results presented in biomedical publications and to facilitate additional exploration of the reported data. However, there is a lack of convenient tools that balance reproducibility with ease of use. To address this problem, we develop BioVisReport, a lightweight solution for the rapid generation of an interactive website based on a user-defined Markdown file, which acts as a text markup language without requiring users to master complex syntax and allows them to preview the results in real-time. Interactive websites generated by the tool can help readers conveniently reproduce research findings and perform further in-depth analyses beyond those reported in the original peer-reviewed publications. Currently, BioVisReport offers 17 basic types of plots for visualizing published data. In addition, the extensibility of BioVisReport supports flexible integration of user-developed Python plugins with multiple programming languages. BioVisReport is freely available at https://biovis.report/.

GeneTonic: an R/Bioconductor package for streamlining the interpretation of RNA-seq data

BACKGROUND

The interpretation of results from transcriptome profiling experiments via RNA sequencing (RNA-seq) can be a complex task, where the essential information is distributed among different tabular and list formats-normalized expression values, results from differential expression analysis, and results from functional enrichment analyses. A number of tools and databases are widely used for the purpose of identification of relevant functional patterns, yet often their contextualization within the data and results at hand is not straightforward, especially if these analytic components are not combined together efficiently.

RESULTS

We developed the GeneTonic software package, which serves as a comprehensive toolkit for streamlining the interpretation of functional enrichment analyses, by fully leveraging the information of expression values in a differential expression context. GeneTonic is implemented in R and Shiny, leveraging packages that enable HTML-based interactive visualizations for executing drilldown tasks seamlessly, viewing the data at a level of increased detail. GeneTonic is integrated with the core classes of existing Bioconductor workflows, and can accept the output of many widely used tools for pathway analysis, making this approach applicable to a wide range of use cases. Users can effectively navigate interlinked components (otherwise available as flat text or spreadsheet tables), bookmark features of interest during the exploration sessions, and obtain at the end a tailored HTML report, thus combining the benefits of both interactivity and reproducibility.

CONCLUSION

GeneTonic is distributed as an R package in the Bioconductor project ( https://bioconductor.org/packages/GeneTonic/ ) under the MIT license. Offering both bird's-eye views of the components of transcriptome data analysis and the detailed inspection of single genes, individual signatures, and their relationships, GeneTonic aims at simplifying the process of interpretation of complex and compelling RNA-seq datasets for many researchers with different expertise profiles.

The new normal? Redaction bias in biomedical science

A concerning amount of biomedical research is not reproducible. Unreliable results impede empirical progress in medical science, ultimately putting patients at risk. Many proximal causes of this irreproducibility have been identified, a major one being inappropriate statistical methods and analytical choices by investigators. Within this, we formally quantify the impact of inappropriate redaction beyond a threshold value in biomedical science. This is effectively truncation of a dataset by removing extreme data points, and we elucidate its potential to accidentally or deliberately engineer a spurious result in significance testing. We demonstrate that the removal of a surprisingly small number of data points can be used to dramatically alter a result. It is unknown how often redaction bias occurs in the broader literature, but given the risk of distortion to the literature involved, we suggest that it must be studiously avoided, and mitigated with approaches to counteract any potential malign effects to the research quality of medical science.

maplet: an extensible R toolbox for modular and reproducible metabolomics pipelines

This article presents maplet, an open-source R package for the creation of highly customizable, fully reproducible statistical pipelines for metabolomics data analysis. It builds on the SummarizedExperiment data structure to create a centralized pipeline framework for storing data, analysis steps, results and visualizations. maplet's key design feature is its modularity, which offers several advantages, such as ensuring code quality through the maintenance of individual functions and promoting collaborative development by removing technical barriers to code contribution. With over 90 functions, the package includes a wide range of functionalities, covering many widely used statistical approaches and data visualization techniques.

AVAILABILITY AND IMPLEMENTATION

The maplet package is implemented in R and freely available at https://github.com/krumsieklab/maplet.

Enhancing research informatics core user satisfaction through agile practices

OBJECTIVE

The Huntsman Cancer Institute Research Informatics Shared Resource (RISR), a software and database development core facility, sought to address a lack of published operational best practices for research informatics cores. It aimed to use those insights to enhance effectiveness after an increase in team size from 20 to 31 full-time equivalents coincided with a reduction in user satisfaction.

MATERIALS AND METHODS

RISR migrated from a water-scrum-fall model of software development to agile software development practices, which emphasize iteration and collaboration. RISR's agile implementation emphasizes the product owner role, which is responsible for user engagement and may be particularly valuable in software development that requires close engagement with users like in science.

RESULTS

All RISR's software development teams implemented agile practices in early 2020. All project teams are led by a product owner who serves as the voice of the user on the development team. Annual user survey scores for service quality and turnaround time recorded 9 months after implementation increased by 17% and 11%, respectively.

DISCUSSION

RISR is illustrative of the increasing size of research informatics cores and the need to identify best practices for maintaining high effectiveness. Agile practices may address concerns about the fit of software engineering practices in science. The study had one time point after implementing agile practices and one site, limiting its generalizability.

CONCLUSIONS

Agile software development may substantially increase a research informatics core facility's effectiveness and should be studied further as a potential best practice for how such cores are operated.

The role of metadata in reproducible computational research

Reproducible computational research (RCR) is the keystone of the scientific method for in silico analyses, packaging the transformation of raw data to published results. In addition to its role in research integrity, improving the reproducibility of scientific studies can accelerate evaluation and reuse. This potential and wide support for the FAIR principles have motivated interest in metadata standards supporting reproducibility. Metadata provide context and provenance to raw data and methods and are essential to both discovery and validation. Despite this shared connection with scientific data, few studies have explicitly described how metadata enable reproducible computational research. This review employs a functional content analysis to identify metadata standards that support reproducibility across an analytic stack consisting of input data, tools, notebooks, pipelines, and publications. Our review provides background context, explores gaps, and discovers component trends of embeddedness and methodology weight from which we derive recommendations for future work.

Metabolism-based therapies for epilepsy: new directions for future cures

OBJECTIVE

Thousands of years after dietary therapy was proposed to treat seizures, how alterations in metabolism relates to epilepsy remains unclear, and metabolism-based therapies are not always effective.

METHODS

We consider the state of the science in metabolism-based therapies for epilepsy across the research lifecycle from basic to translational to clinical studies.

RESULTS

This analysis creates a conceptual framework for creative, rigorous, and transparent research to benefit people with epilepsy through the understanding and modification of metabolism.

INTERPRETATION

Despite intensive past efforts to evaluate metabolism-based therapies for epilepsy, distinct ways of framing a problem offer the chance to engage different mindsets and new (or newly applied) technologies. A comprehensive, creative, and inclusive problem-directed research agenda is needed, with a renewed and stringent adherence to rigor and transparency across all levels of investigation.

Fostering reproducibility, reusability, and technology transfer in health informatics

Computational methods can transform healthcare. In particular, health informatics with artificial intelligence has shown tremendous potential when applied in various fields of medical research and has opened a new era for precision medicine. The development of reusable biomedical software for research or clinical practice is time-consuming and requires rigorous compliance with quality requirements as defined by international standards.

However, research projects rarely implement such measures, hindering smooth technology transfer into the research community or manufacturers as well as reproducibility and reusability.

Here, we present a guideline for quality management systems (QMS) for academic organizations incorporating the essential components while confining the requirements to an easily manageable effort. It provides a starting point to implement a QMS tailored to specific needs effortlessly and greatly facilitates technology transfer in a controlled manner, thereby supporting reproducibility and reusability.

Ultimately, the emerging standardized workflows can pave the way for an accelerated deployment in clinical practice.

Easyreporting simplifies the implementation of Reproducible Research layers in R software

During last years "irreproducibility" became a general problem in omics data analysis due to the use of sophisticated and poorly described computational procedures. For avoiding misleading results, it is necessary to inspect and reproduce the entire data analysis as a unified product. Reproducible Research (RR) provides general guidelines for public access to the analytic data and related analysis code combined with natural language documentation, allowing third-parties to reproduce the findings. We developed easyreporting, a novel R/Bioconductor package, to facilitate the implementation of an RR layer inside reports/tools. We describe the main functionalities and illustrate the organization of an analysis report using a typical case study concerning the analysis of RNA-seq data. Then, we show how to use easyreporting in other projects to trace R functions automatically. This latter feature helps developers to implement procedures that automatically keep track of the analysis steps. Easyreporting can be useful in supporting the reproducibility of any data analysis project and shows great advantages for the implementation of R packages and GUIs. It turns out to be very helpful in bioinformatics, where the complexity of the analyses makes it extremely difficult to trace all the steps and parameters used in the study.

Understanding experiments and research practices for reproducibility: an exploratory study

Scientific experiments and research practices vary across disciplines. The research practices followed by scientists in each domain play an essential role in the understandability and reproducibility of results. The "Reproducibility Crisis", where researchers find difficulty in reproducing published results, is currently faced by several disciplines. To understand the underlying problem in the context of the reproducibility crisis, it is important to first know the different research practices followed in their domain and the factors that hinder reproducibility. We performed an exploratory study by conducting a survey addressed to researchers representing a range of disciplines to understand scientific experiments and research practices for reproducibility. The survey findings identify a reproducibility crisis and a strong need for sharing data, code, methods, steps, and negative and positive results. Insufficient metadata, lack of publicly available data, and incomplete information in study methods are considered to be the main reasons for poor reproducibility. The survey results also address a wide number of research questions on the reproducibility of scientific results. Based on the results of our explorative study and supported by the existing published literature, we offer general recommendations that could help the scientific community to understand, reproduce, and reuse experimental data and results in the research data lifecycle.

GeenaR: A Web Tool for Reproducible MALDI-TOF Analysis

Mass spectrometry is a widely applied technology with a strong impact in the proteomics field. MALDI-TOF is a combined technology in mass spectrometry with many applications in characterizing biological samples from different sources, such as the identification of cancer biomarkers, the detection of food frauds, the identification of doping substances in athletes’ fluids, and so on. The massive quantity of data, in the form of mass spectra, are often biased and altered by different sources of noise. Therefore, extracting the most relevant features that characterize the samples is often challenging and requires combining several computational methods. Here, we present GeenaR, a novel web tool that provides a complete workflow for pre-processing, analyzing, visualizing, and comparing MALDI-TOF mass spectra. GeenaR is user-friendly, provides many different functionalities for the analysis of the mass spectra, and supports reproducible research since it produces a human-readable report that contains function parameters, results, and the code used for processing the mass spectra. First, we illustrate the features available in GeenaR. Then, we describe its internal structure. Finally, we prove its capabilities in analyzing oncological datasets by presenting two case studies related to ovarian cancer and colorectal cancer. GeenaR is available at http://proteomics.hsanmartino.it/geenar/.

Subtypes of Preeclampsia: Recognition and Determining Clinical Usefulness

The concept that preeclampsia is a multisystemic syndrome is appreciated in both research and clinical care. Our understanding of pathophysiology recognizes the role of inflammation, oxidative and endoplasm reticulum stress, and angiogenic dysfunction. Yet, we have not progressed greatly toward clinically useful prediction nor had substantial success in prevention or treatment. One possibility is that the maternal syndrome may be reached through different pathophysiological pathways, that is, subtypes of preeclampsia, that in their specificity yield more clinical utility. For example, early and late onset preeclampsia are increasingly acknowledged as different pathophysiological processes leading to a common presentation. Other subtypes of preeclampsia are supported by disparate clinical outcomes, long-range prognosis, organ systems involved, and risk factors. These insights have been supplemented by discovery-driven methods, which cluster preeclampsia cases into groups indicating different pathophysiologies. In this presentation, we review likely subtypes based on current knowledge and suggest others. We present a consideration of the requirements for a clinically meaningful preeclampsia subtype. A useful subtype should (1) identify a specific pathophysiological pathway or (2) specifically indicate maternal or fetal outcome, (3) be recognizable in a clinically useful time frame, and (4) these results should be reproducible and generalizable (but at varying frequency) including in low resource settings. We recommend that the default consideration be that preeclampsia includes several subtypes rather than trying to force all cases into a single pathophysiological pathway. The recognition of subtypes and deciphering their different pathophysiologies will provide specific targets for prevention, prediction, and treatment directing personalized care.

The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment

Objective

Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers.

Materials and Methods

The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics.

Results

Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access.

Conclusions

The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

Ten simple rules for writing a paper about scientific software

Papers describing software are an important part of computational fields of scientific research. These “software papers” are unique in a number of ways, and they require special consideration to improve their impact on the scientific community and their efficacy at conveying important information. Here, we discuss 10 specific rules for writing software papers, covering some of the different scenarios and publication types that might be encountered, and important questions from which all computational researchers would benefit by asking along the way.

Computational researchers have a responsibility to ensure that the software they write stands up to the same scientific scrutiny as traditional research studies. These 10 simple rules make doing so easier by enhancing usability, reproducibility, transparency, and other crucial characteristics that aren’t taught in most computer science or research methods curricula.