Characterizing scientific failure: Putting the replication crisis in context

Thinking process templates for constructing data stories with SCDNEY

Background

Globally, scientists now have the ability to generate a vast amount of high throughput biomedical data that carry critical information for important clinical and public health applications. This data revolution in biology is now creating a plethora of new single-cell datasets. Concurrently, there have been significant methodological advances in single-cell research. Integrating these two resources, creating tailor-made, efficient, and purpose-specific data analysis approaches can assist in accelerating scientific discovery.

Methods

We developed a series of living workshops for building data stories, using Single-cell data integrative analysis (scdney). scdney is a wrapper package with a collection of single-cell analysis R packages incorporating data integration, cell type annotation, higher order testing and more.

Results

Here, we illustrate two specific workshops. The first workshop examines how to characterise the identity and/or state of cells and the relationship between them, known as phenotyping. The second workshop focuses on extracting higher-order features from cells to predict disease progression.

Conclusions

Through these workshops, we not only showcase current solutions, but also highlight critical thinking points. In particular, we highlight the Thinking Process Template that provides a structured framework for the decision-making process behind such single-cell analyses. Furthermore, our workshop will incorporate dynamic contributions from the community in a collaborative learning approach, thus the term 'living'.

On the Role of Theory and Modeling in Neuroscience

In recent years, the field of neuroscience has gone through rapid experimental advances and a significant increase in the use of quantitative and computational methods. This growth has created a need for clearer analyses of the theory and modeling approaches used in the field. This issue is particularly complex in neuroscience because the field studies phenomena that cross a wide range of scales and often require consideration at varying degrees of abstraction, from precise biophysical interactions to the computations they implement. We argue that a pragmatic perspective of science, in which descriptive, mechanistic, and normative models and theories each play a distinct role in defining and bridging levels of abstraction, will facilitate neuroscientific practice. This analysis leads to methodological suggestions, including selecting a level of abstraction that is appropriate for a given problem, identifying transfer functions to connect models and data, and the use of models themselves as a form of experiment.

Generalization Bias in Science

Many scientists routinely generalize from study samples to larger populations. It is commonly assumed that this cognitive process of scientific induction is a voluntary inference in which researchers assess the generalizability of their data and then draw conclusions accordingly. We challenge this view and argue for a novel account. The account describes scientific induction as involving by default a generalization bias that operates automatically and frequently leads researchers to unintentionally generalize their findings without sufficient evidence. The result is unwarranted, overgeneralized conclusions. We support this account of scientific induction by integrating a range of disparate findings from across the cognitive sciences that have until now not been connected to research on the nature of scientific induction. The view that scientific induction involves by default a generalization bias calls for a revision of the current thinking about scientific induction and highlights an overlooked cause of the replication crisis in the sciences. Commonly proposed interventions to tackle scientific overgeneralizations that may feed into this crisis need to be supplemented with cognitive debiasing strategies against generalization bias to most effectively improve science.

Dynamic publication media with the COPASI R Connector (CoRC)

In this article we show how dynamic publication media and the COPASI R Connector (CoRC) can be combined in a natural and synergistic way to communicate (biochemical) models. Dynamic publication media are becoming a popular tool for authors to effectively compose and publish their work. They are built from templates and the final documents are created dynamically. In addition, they can also be interactive. Working with dynamic publication media is made easy with the programming environment R via its integration with tools such as R Markdown, Jupyter and Shiny. Additionally, the COmplex PAthway SImulator COPASI (http://www.copasi.org), a widely used biochemical modelling toolkit, is available in R through the use of the COPASI R Connector (CoRC, https://jpahle.github.io/CoRC). Models are a common tool in the mathematical biosciences, in particular kinetic models of biochemical networks in (computational) systems biology. We focus on three application areas of dynamic publication media and CoRC: Documentation (reproducible workflows), Teaching (creating self-paced lessons) and Science Communication (immersive and engaging presentation). To illustrate these, we created six dynamic document examples in the form of R Markdown and Jupyter notebooks, hosted on the platforms GitHub, shinyapps.io, Google Colaboratory. Having code and output in one place, creating documents in template-form and the option of interactivity make the combination of dynamic documents and CoRC a versatile tool. All our example documents are freely available at https://jpahle.github.io/DynamiCoRC under the Creative Commons BY 4.0 licence.

Self-correction in science: The diagnostic and integrative motives for replication

A series of failed replications and frauds have raised questions regarding self-correction in science. Metascientific activists have advocated policies that incentivize replications and make them more diagnostically potent. We argue that current debates, as well as research in science and technology studies, have paid little heed to a key dimension of replication practice. Although it sometimes serves a diagnostic function, replication is commonly motivated by a practical desire to extend research interests. The resulting replication, which we label 'integrative', is characterized by a pragmatic flexibility toward protocols. The goal is to appropriate what is useful, not test for truth. Within many experimental cultures, however, integrative replications can produce replications of ambiguous diagnostic power. Based on interviews with 60 members of the Board of Reviewing Editors for the journal Science, we show how the interplay between the diagnostic and integrative motives for replication differs between fields and produces different cultures of replication. We offer six theses that aim to put science and technology studies and science activism into dialog to show why effective reforms will need to confront issues of disciplinary difference.

The thin ret(raction) line: biomedical journal responses to incorrect non-targeting nucleotide sequence reagents in human gene knockdown publications

The capacity of the scientific literature to self-correct is of vital importance, but few studies have compared post-publication journal responses to specific error types. We have compared journal responses to a specific reagent error in 31 human gene knockdown publications, namely a non-targeting or negative control nucleotide sequence that is instead predicted to target a human gene. The 31 papers published by 13 biomedical journals generated 26 published responses (14 retractions, 5 expressions of concern, 7 author corrections which included one resolved expression of concern) as well as 6 stated decisions to take no action. Variations in published responses were noted both between journals and by 4 journals that published different responses to at least 2 papers. A subset of published responses revealed conflicting explanations for the wrongly identified control reagent, despite 30/31 papers obtaining their gene knockdown reagents from the same external supplier. Viewed collectively, different journal responses to human gene knockdown publications with a common reagent error type suggest that editorial staff require more support to interpret post-publication notifications of incorrect nucleotide sequence reagents. We propose a draft template to facilitate the communication, interpretation and investigation of published errors, including errors affecting research reagents.

Leveraging preclinical models for the development of Alzheimer disease therapeutics

A large number of mouse models have been engineered, characterized and used to advance biomedical research in Alzheimer disease (AD). Early models simply damaged the rodent brain through toxins or lesions. Later, the spread of genetic engineering technology enabled investigators to develop models of familial AD by overexpressing human genes such as those encoding amyloid precursor protein (APP) or presenilins (PSEN1 or PSEN2) carrying mutations linked to early-onset AD. Recently, more complex models have sought to explore the impact of multiple genetic risk factors in the context of different biological challenges. Although none of these models has proven to be a fully faithful reproduction of the human disease, models remain essential as tools to improve our understanding of AD biology, conduct thorough pharmacokinetic and pharmacodynamic analyses, discover translatable biomarkers and evaluate specific therapeutic approaches. To realize the full potential of animal models as new technologies and knowledge become available, it is critical to define an optimal strategy for their use. Here, we review progress and challenges in the use of AD mouse models, highlight emerging scientific innovations in model development, and introduce a conceptual framework for use of preclinical models for therapeutic development.