Can the behavioral sciences self-correct? A social epistemic study

Short-term auditory priming in freely-moving mice

Priming, a change in the mental processing of a stimulus as a result of prior encounter with a related stimulus, has been observed repeatedly and studied extensively in humans. Yet currently, there is no behavioral model of short-term priming in lab animals, precluding research on the neurobiological basis of priming. Here, we describe an auditory discrimination paradigm for studying response priming in freely moving mice. We find a priming effect in success rate in all mice tested on the task. In contrast, we do not find a priming effect in response times. Compared to non-primed discrimination trials, the addition of incongruent prime stimuli reduces success rate more than congruent prime stimuli, suggesting a cognitive mechanism based on differential interference. The results establish the short-term priming phenomenon in rodents, and the paradigm opens the door to studying the cellular-network basis of priming.

Open science, the replication crisis, and environmental public health

Concerns about a crisis of mass irreplicability across scientific fields ("the replication crisis") have stimulated a movement for open science, encouraging or even requiring researchers to publish their raw data and analysis code. Recently, a rule at the US Environmental Protection Agency (US EPA) would have imposed a strong open data requirement. The rule prompted significant public discussion about whether open science practices are appropriate for fields of environmental public health. The aims of this paper are to assess (1) whether the replication crisis extends to fields of environmental public health; and (2) in general whether open science requirements can address the replication crisis. There is little empirical evidence for or against mass irreplicability in environmental public health specifically. Without such evidence, strong claims about whether the replication crisis extends to environmental public health - or not - seem premature. By distinguishing three concepts - reproducibility, replicability, and robustness - it is clear that open data initiatives can promote reproducibility and robustness but do little to promote replicability. I conclude by reviewing some of the other benefits of open science, and offer some suggestions for funding streams to mitigate the costs of adoption of open science practices in environmental public health.

Publish without bias or perish without replications

There is mounting evidence that a large portion of experimental results cannot be replicated, leading many to believe that science is now in the throes of a replicability crisis. In response, there have been calls to reduce publication bias against negative results because of the effect that publication bias has on the publication record. Others, however, argue that publication bias need not be detrimental to scientific progress. Here, we propose a novel mechanism by dint of which reducing publication bias can benefit science regardless of the effect that publication bias has on the publication record. To do so, we introduce a series of increasingly complex mathematical models. Our models represent a scientific community consisting of discovery researchers who test novel hypotheses, and confirmation researchers who test known hypotheses. Results show that reducing publication bias can have the surprising consequence of increasing the share of confirmation researchers who conduct replications. When a large share of scientists conduct confirmation research, scientists have an incentive to conduct high-quality research as others are likely to check their findings. Our models therefore suggest an underappreciated reasons why reducing publication bias might benefit science.

Objectivity for the research worker

In the last decade, many problematic cases of scientific conduct have been diagnosed; some of which involve outright fraud (e.g., Stapel, 2012) others are more subtle (e.g., supposed evidence of extrasensory perception; Bem, 2011). These and similar problems can be interpreted as caused by lack of scientific objectivity. The current philosophical theories of objectivity do not provide scientists with conceptualizations that can be effectively put into practice in remedying these issues. We propose a novel way of thinking about objectivity for individual scientists; a negative and dynamic approach.We provide a philosophical conceptualization of objectivity that is informed by empirical research. In particular, it is our intention to take the first steps in providing an empirically and methodologically informed inventory of factors that impair the scientific practice. The inventory will be compiled into a negative conceptualization (i.e., what is not objective), which could in principle be used by individual scientists to assess (deviations from) objectivity of scientific practice. We propose a preliminary outline of a usable and testable instrument for indicating the objectivity of scientific practice.

Scientific conclusions need not be accurate, justified, or believed by their authors

We argue that the main results of scientific papers may appropriately be published even if they are false, unjustified, and not believed to be true or justified by their author. To defend this claim we draw upon the literature studying the norms of assertion, and consider how they would apply if one attempted to hold claims made in scientific papers to their strictures, as assertions and discovery claims in scientific papers seem naturally analogous. We first use a case study of William H. Bragg's early twentieth century work in physics to demonstrate that successful science has in fact violated these norms. We then argue that features of the social epistemic arrangement of science which are necessary for its long run success require that we do not hold claims of scientific results to their standards. We end by making a suggestion about the norms that it would be appropriate to hold scientific claims to, along with an explanation of why the social epistemology of science-considered as an instance of collective inquiry-would require such apparently lax norms for claims to be put forward.

E-Synthesis: A Bayesian Framework for Causal Assessment in Pharmacosurveillance

Background: Evidence suggesting adverse drug reactions often emerges unsystematically and unpredictably in form of anecdotal reports, case series and survey data. Safety trials and observational studies also provide crucial information regarding the (un-)safety of drugs. Hence, integrating multiple types of pharmacovigilance evidence is key to minimising the risks of harm.

Methods: In previous work, we began the development of a Bayesian framework for aggregating multiple types of evidence to assess the probability of a putative causal link between drugs and side effects. This framework arose out of a philosophical analysis of the Bradford Hill Guidelines. In this article, we expand the Bayesian framework and add “evidential modulators,” which bear on the assessment of the reliability of incoming study results. The overall framework for evidence synthesis, “E-Synthesis”, is then applied to a case study.

Results: Theoretically and computationally, E-Synthesis exploits coherence of partly or fully independent evidence converging towards the hypothesis of interest (or of conflicting evidence with respect to it), in order to update its posterior probability. With respect to other frameworks for evidence synthesis, our Bayesian model has the unique feature of grounding its inferential machinery on a consolidated theory of hypothesis confirmation (Bayesian epistemology), and in allowing any data from heterogeneous sources (cell-data, clinical trials, epidemiological studies), and methods (e.g., frequentist hypothesis testing, Bayesian adaptive trials, etc.) to be quantitatively integrated into the same inferential framework.

Conclusions: E-Synthesis is highly flexible concerning the allowed input, while at the same time relying on a consistent computational system, that is philosophically and statistically grounded. Furthermore, by introducing evidential modulators, and thereby breaking up the different dimensions of evidence (strength, relevance, reliability), E-Synthesis allows them to be explicitly tracked in updating causal hypotheses.

Who Should Do Replication Labor?

Scientists, for the most part, want to get it right. However, the social structures that govern their work undermine that aim, and this leads to nonreplicable findings in many fields. Because the social structure of science is a decentralized system, it is difficult to intervene. In this article, I discuss how we might do so, focusing on self-corrective-labor schemes (i.e., ways of distributing replication efforts within the scientific community). First, I argue that we need to implement a scheme that makes replication work outcome independent, systematic, and sustainable. Second, I use these three criteria to evaluate extant proposals, which place the responsibility for replication on original researchers, consumers of their research, students, or many labs. Third, on the basis of a philosophical analysis of the reward system of science and the benefits of the division of cognitive labor, I propose a scheme that satisfies the criteria better: the professional scheme. This scheme has two main components. First, the scientific community is organized into two groups: discovery researchers, who produce new findings, and confirmation researchers, whose primary function is to do confirmation work (i.e., replication, reproduction, meta-analysis). Second, a distinct reward system is established for confirmation researchers so that their career advancement is separated from whether they obtain positive experimental results.