Developing an open science 'mindset'

BACKGROUND

Identification of widespread biases present in reported research findings in many scientific disciplines, including psychology, such as failures to replicate and the likely extensive application of questionable research practices, has raised serious concerns over the reliability and trustworthiness of scientific research. This has led to the development of, and advocacy for, 'open science' practices, including data, materials, analysis, and output sharing, pre-registration of study predictions and analysis plans, and increased access to published research findings. Implementation of such practices has been enthusiastic in some quarters, but literacy in, and adoption of, these practices has lagged behind among many researchers in the scientific community.

ADVANCES

In the current article I propose that researchers adopt an open science 'mindset', a comprehensive approach to open science predicated on researchers' operating under the basic assumption that, wherever possible, open science practices will be a central component of all steps of their research projects. The primary, defining feature of the mindset is a commitment to open science principles in all research projects from inception to dissemination. Other features of the mindset include the assumption that all components of research projects (e.g. pre-registered hypotheses, protocols, materials, analysis plans, data, and output) will be accessible broadly; pro-active selection of open fora to disseminate research components and findings; open and transparent dissemination of reports of the research findings in advance of, and after, formal publication; and active promotion of open science practices through education, modeling, and advocacy.

CONCLUSION

The open science mindset is a 'farm to fork' approach to open science aimed at promoting comprehensive quality in application of open science, and widening participation in open science practices so that they become the norm in research in health psychology and behavioral medicine going forward.

Manipulation of the Duration of the Initial Self-Control Task Within the Sequential-Task Paradigm: Effect on Exercise Performance

Self-control exertion on an initial task has been associated with impaired performance on subsequent physical tasks also requiring self-control; an effect suggested to be mediated by changes in perceptions of pain and motivation. However, the effects of spending longer on the initial self-control task are unknown. This study, therefore, explored the potential for the duration of the initial self-control task to influence subsequent physical performance, perceptions of pain, and perceived motivation; particularly during the early stages of the physical task. In a within-subject design, 29 participants (11 male, 18 female) completed a wall-sit task until volitional exhaustion, on four separate occasions. Prior to each wall-sit, participants completed either a non-self-control task (congruent Stroop task) for 4 min, or a self-control task (incongruent Stroop task) for 4 (short duration), 8 (medium duration), or 16 (long duration) min. Participant's perceptions of pain and motivation were recorded every 30 s during the wall-sit. Wall-sit performance time was analyzed using one-way ANOVA and perceptions of pain and motivation analyzed using multi-level modeling. Wall-sit performance time was significantly longer on the non-self-control exertion trial compared to all other trials (all p < 0.01), as well as longer on both the short duration and medium duration self-control exertion trials compared to the long duration self-control exertion trial (both p < 0.001). Perceptions of initial (at 30 s) pain and motivation were different between the trials (main effect of trial: pain, p = 0.001; motivation, p < 0.001); whereby longer durations of self-control exertion increased perceptions of pain and decreased motivation. The decrease in motivation during the wall-sit task was greater on the long duration self-control exertion trial compared to all other trials (trial^∗time interactions, all p < 0.05). The present study provides novel evidence that spending longer on the initial self-control task led to greater detrimental effects on subsequent wall-sit performance time. Furthermore, longer duration self-control exertion tasks led to increased perceptions of pain and decreased motivation within the first 30 s of the wall-sit task, as well as a greater decrease in motivation across the wall-sit task. These attentional and motivational shifts may explain performance decrements following the exertion of self-control.

The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture

Like many other areas of science, experimental psychology is affected by a "replication crisis" that is causing concern in many fields of research. Approaches to tackling this crisis include better training in statistical methods, greater transparency and openness, and changes to the incentives created by funding agencies, journals, and institutions. Here, I argue that if proposed solutions are to be effective, we also need to take into account human cognitive constraints that can distort all stages of the research process, including design and execution of experiments, analysis of data, and writing up findings for publication. I focus specifically on cognitive schemata in perception and memory, confirmation bias, systematic misunderstanding of statistics, and asymmetry in moral judgements of errors of commission and omission. Finally, I consider methods that may help mitigate the effect of cognitive constraints: better training, including use of simulations to overcome statistical misunderstanding; specific programmes directed at inoculating against cognitive biases; adoption of Registered Reports to encourage more critical reflection in planning studies; and using methods such as triangulation and "pre mortem" evaluation of study design to foster a culture of dialogue and criticism.

The Limits of Ego Depletion

Abstract. Evidence on the existence of the ego depletion phenomena as well as the size of the effects and potential moderators and mediators are ambiguous. Building on a crossover design that enables superior statistical power within a single study, we investigated the robustness of the ego depletion effect between and within subjects and moderating and mediating influences of the ego depletion manipulation checks. Our results, based on a sample of 187 participants, demonstrated that (a) the between- and within-subject ego depletion effects only had negligible effect sizes and that there was (b) large interindividual variability that (c) could not be explained by differences in ego depletion manipulation checks. We discuss the implications of these results and outline a future research agenda.

A Validation Study of the German Complex-Span Tasks and Some General Considerations on Task Translation Procedures in Cognitive Psychology

Abstract. Automated complex-span tasks are widely used to assess working-memory capacity and the English versions show good psychometric properties ( Unsworth, Heitz, Schrock, & Engle, 2005 ). However, it is generally an open question whether translated task versions have the same properties as the original versions and whether results obtained with translated tasks can be interpreted equivalently to those obtained with the original tasks. We translated the complex-span tasks and had a sample of German participants perform these tasks as well as a running-memory-span task and a reasoning test. We assessed the reliabilities of the German complex-span tasks and their construct and criterion-related validities. Extrapolating from cross-cultural literature, we also employed a test of measurement invariance to compare the correlational patterns as well as the construct structure between the German sample and a similar North-American sample. Results show that the German complex-span tasks are reliable and valid indicators of working-memory capacity and that they are metrically and functionally equivalent to the original versions. As measurement equivalence is an important but often neglected topic in basic cognitive psychology, we also highlight the general benefits of using equivalence tests when translating cognitive tasks.

Task duration and task order do not matter: no effect on self-control performance

The strength model of self-control proposes that all acts of self-control are energized by one global limited resource that becomes temporarily depleted by a primary self-control task, leading to impaired self-control performance in secondary self-control tasks. However, failed replications have cast doubt on the existence of this so-called ego depletion effect. Here, we investigated between-task (i.e., variation in self-control tasks) and within-task variation (i.e., task duration) as possible explanations for the conflicting literature on ego depletion effects. In a high-powered experiment (N = 709 participants), we used two established self-control tasks (Stroop task, transcription task) to test how variations in the duration of primary and secondary self-control tasks (2, 4, 8, or 16 min per task) affect the occurrence of an ego depletion effect (i.e., impaired performance in the secondary task). In line with the ego depletion hypothesis, subjects perceived longer lasting secondary tasks as more self-control demanding. Contrary to the ego depletion hypothesis, however, performance did neither suffer from prior self-control exertion, nor as a function of task duration. If anything, performance tended to improve when the primary self-control task lasted longer. These effects did not differ between the two self-control tasks, suggesting that the observed null findings were independent of task type.

Self-reports from behind the scenes: Questionable research practices and rates of replication in ego depletion research

The strength model of self-control is one of the most influential and well-established models of self-regulation in social psychology. However, recent attempts to replicate the ego depletion effect have sometimes failed. The goal of this study is to investigate self-reported replication rates and the frequency of a set of questionable research practices (QRP) in ego depletion research. A literature search resulted in 1721 researchers who had previously published on ego depletion. They were invited to participate in an anonymous online survey. The respondents (n = 277), on average, had published over three papers on ego depletion, and had completed more than two additional, unpublished studies. Respondents indicated that in more than 40% of their studies, results were similar in magnitude to those reported in the existing literature, and more than 60% reported conducting a priori power analyses. 39.2% of respondents were aware of other researchers who engaged in the surveyed QRP's, while 37.7% affirmed to have employed said QRP's. These results underline the importance of reducing QRP's to reliably test the validity of the ego depletion effect.

The (non-)replicability of regulatory resource depletion: A field report employing non-invasive brain stimulation

Cognitive effort and self-control are exhausting. Although evidence is ambiguous, behavioural studies have repeatedly suggested that control-demanding tasks seem to deplete a limited cache of self-regulatory resources leading to performance degradations and fatigue. While resource depletion has indirectly been associated with a decline in right prefrontal cortex capacity, its precise neural underpinnings have not yet been revealed. This study consisted of two independent experiments, which set out to investigate the causal role of the right dorsolateral prefrontal cortex (DLPFC) in a classic dual phase depletion paradigm employing non-invasive brain stimulation. In Experiment 1 we demonstrated a general depletion effect, which was significantly eliminated by anodal transcranial Direct Current Stimulation to the right DLPFC. In Experiment 2, however, we failed to replicate the basic psychological depletion effect within a second independent sample. The dissimilar results are discussed in the context of the current 'replication crisis' and suggestions for future studies are offered. While our current results do not allow us to firmly argue for or against the existence of resource depletion, we outline why it is crucial to further clarify which specific external and internal circumstances lead to limited replicability of the described effect. We showcase and discuss the current inter-lab replication problem based on two independent samples tested within one research group (intra-lab).