Prescription for Love: An Experimental Investigation of Laypeople's Relative Moral Disapproval of Love Drugs

New technologies regularly bring about profound changes in our daily lives. Romantic relationships are no exception to these transformations. Some philosophers expect the emergence in the near future of love drugs: a theoretically achievable biotechnological intervention that could be designed to strengthen and maintain love in romantic relationships. We investigated laypeople's resistance to the use of such technologies and its sources. Across two studies (Study 1, French and Peruvian university students, N after exclusion = 186; Study 2, Amazon Mechanical Turk sample, N after exclusion = 693, pre-registered), we found that the use of love drugs designed to strengthen and maintain love in romantic relationships are considered as more morally problematic than psychological therapy with the same aim. In Study 2, we show that this last effect is partially due to the fact that the love resulting from the use of love drugs is perceived as less authentic, intense, and durable. We discuss the specific role of authenticity in the relative moral disapproval of love drugs.

"When" Versus "Whether" Gender/Sex Differences: Insights From Psychological Research on Negotiation, Risk-Taking, and Leadership

We present a conceptual framework of situational moderators of gender/sex effects in negotiation, risk-taking, and leadership-three masculine-stereotypic domains associated with gender/sex gaps in pay and authority. We propose that greater situational ambiguity and higher relevance and salience of gender/sex increase the likelihood of gender/sex-linked behaviors in these domains. We argue that greater ambiguity increases the extent to which actors and audiences must search inwardly (e.g., mental schema, past experience) or outwardly (e.g., social norms) for cues on how to behave or evaluate a situation and thereby widens the door for gender/sex-linked influences. Correspondingly, we propose that gender/sex effects on behavior and evaluations in these domains will be more likely when gender/sex is more relevant and salient to the setting or task. We propose further that these two situational moderators may work jointly or interactively to influence the likelihood of gender/sex effects in negotiation, risk-taking, and leadership. We conclude by discussing applications of our conceptual framework to psychological science and its translation to practice, including directions for future research.

Contamination-Focussed Vignettes as an Analogue of Infectious Pandemics: An Experimental Validation using the State Disgust and Anxiety Responses in OCD

Despite infectious pandemics proving particularly detrimental to those with Obsessive-Compulsive Disorder (OCD), the investigation of analogous experimental paradigms is lacking. To address this gap, we conducted two studies employing vignettes that depicted contamination-related situations commonly experienced during a pandemic (e.g., Coughing into hands and failing to use hand sanitizer). We manipulated the salience of these vignettes across three levels: high contamination, low contamination, and a neutral control condition. Our examination of state anxiety and disgust responses in all participants revealed the successful manipulation of the vignettes' impact. Specifically, individuals with more severe OCD symptoms reported significantly higher levels of state disgust and anxiety for both high and low contamination vignettes, in contrast to the group with lower symptom severity. No significant differences were observed in the neutral vignette condition between the high- and low-scoring groups. Interestingly, for those with higher OCD symptoms, high salience contamination-focused vignettes resulted in similarly elevated state disgust and anxiety, regardless of whether the vignettes were situated in public (Study 1) or domestic (Study 2) settings. This suggests that the heightened sensitivity to contamination-related scenarios observed in individuals with OCD symptoms in the present study is not confined to a specific context. These findings support the use of contamination-focused vignettes as analogues for studying infectious pandemics and provide valuable insights into OCD models, interventions, and future research.

Exploring a formal approach to selecting studies for replication: A feasibility study in social neuroscience

Replication of published results is crucial for ensuring the robustness and self-correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often have to decide which of several relevant candidates to target for replication. Formal strategies for efficient study selection have been proposed, but none have been explored for practical feasibility - a prerequisite for validation. Here we move one step closer to efficient replication study selection by exploring the feasibility of a particular selection strategy that estimates replication value as a function of citation impact and sample size (Isager, van 't Veer, & Lakens, 2021). We tested our strategy on a sample of fMRI studies in social neuroscience. We first report our efforts to generate a representative candidate set of replication targets. We then explore the feasibility and reliability of estimating replication value for the targets in our set, resulting in a dataset of 1358 studies ranked on their value of prioritising them for replication. In addition, we carefully examine possible measures, test auxiliary assumptions, and identify boundary conditions of measuring value and uncertainty. We end our report by discussing how future validation studies might be designed. Our study demonstrates the importance of investigating how to implement study selection strategies in practice. Our sample and study design can be extended to explore the feasibility of other formal study selection strategies that have been proposed.

No evidence that belief in conspiracy theories is negatively related to attitudes toward transhumanism

Transhumanism is a movement that emphasizes the improvement of the human condition by developing technologies and making them widely available. Conspiracy theories regularly refer to the allegedly transhumanist agenda of elites. We hypothesized that belief in conspiracy theories would be related to more unfavorable attitudes toward the transhumanist movement. We examined this association through two pre-registered studies (based on two French samples, total N after exclusion = 550). We found no evidence of a negative relationship between belief in conspiracy theories and attitudes toward transhumanism. This null result was further corroborated by Bayesian analysis, an equivalence test, and an internal mini meta-analysis. This work plays a precursory role in understanding attitudes toward an international cultural and intellectual movement that continues to grow in popularity and influence.

Developmental psychologists should adopt citizen science to improve generalization and reproducibility

Widespread failures of replication and generalization are, ironically, a scientific triumph, in that they confirm the fundamental metascientific theory that underlies our field. Generalizable and replicable findings require testing large numbers of subjects from a wide range of demographics with a large, randomly-sampled stimulus set, and using a variety of experimental parameters. Because few studies accomplish any of this, meta-scientists predict that findings will frequently fail to replicate or generalize. We argue that to be more robust and replicable, developmental psychology needs to find a mechanism for collecting data at greater scale and from more diverse populations. Luckily, this mechanism already exists: Citizen science, in which large numbers of uncompensated volunteers provide data. While best-known for its contributions to astronomy and ecology, citizen science has also produced major findings in neuroscience and psychology, and increasingly in developmental psychology. We provide examples, address practical challenges, discuss limitations, and compare to other methods of obtaining large datasets. Ultimately, we argue that the range of studies where it makes sense *not* to use citizen science is steadily dwindling.

Registered replication report of the construct validity of the error-related negativity (ERN): A multi-site study of task-specific ERN correlations with internalizing and externalizing symptoms

Intact cognitive control is critical for goal-directed behavior and is widely studied using the error-related negativity (ERN). A common assumption in such studies is that ERNs recorded during different experimental paradigms reflect the same construct or functionally equivalent processes and that ERN is functionally distinct from other error-monitoring event-related brain potentials (ERPs; error positivity [Pe]), other neurophysiological indices of cognitive control (N2), and even other theoretically unrelated indices (visual N1). The present registered report represents a replication-plus-extension study of the psychometric validity of cognitive control ERPs and evaluated the convergent and divergent validity of ERN, Pe, N2, and visual N1 recorded during flanker, Stroop, and Go/no-go tasks. Data from 182 participants were collected from two study sites, and ERP psychometric reliability and validity were evaluated. Findings supported replication of convergent and divergent validity of ERN, Pe, and ΔPe (error minus correct)-these ERPs correlated more with themselves across tasks than with other ERPs measured during the same task. Convergent validity of ΔERN across tasks was not replicated, despite high internal consistency. ERN strongly correlated with N2 at levels similar or higher than those in support of convergent validity for other ERPs, and the present study failed to provide evidence of divergent validity for ERN and Pe from N2 or N1. ERN and ΔERN were unrelated to internalizing or externalizing symptoms. Findings underscore the importance of considering the psychometric validity of ERPs, as it provides a foundation for interpreting and comparing ERPs across tasks and studies.

Eleven years of student replication projects provide evidence on the correlates of replicability in psychology

Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of 170   replications . We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.

Research Into Evidence-Based Psychological Interventions Needs a Stronger Focus on Replicability

Background

It is a precondition for evidence-based practice that research is replicable in a wide variety of clinical settings. Current standards for identifying evidence-based psychological interventions and making recommendations for clinical practice in clinical guidelines include criteria that are relevant for replicability, but a better understanding as well refined definitions of replicability are needed enabling empirical research on this topic. Recent advances on this issue were made in the wider field of psychology and in other disciplines, which offers the opportunity to define and potentially increase replicability also in research on psychological interventions.

Method

This article proposes a research strategy for assessing, understanding, and improving replicability in research on psychological interventions.

Results/Conclusion

First, we establish a replication taxonomy ranging from direct to conceptual replication adapted to the field of research on clinical interventions, propose study characteristics that increase the trustworthiness of results, and define statistical criteria for successful replication with respect to the quantitative outcomes of the original and replication studies. Second, we propose how to establish such standards for future research, i.e., in order to design future replication studies for psychological interventions as well as to apply them when investigating which factors are causing the (non-)replicability of findings in the current literature.

Evaluating the Evidence for Enclothed Cognition: Z-Curve and Meta-Analyses

Enclothed cognition refers to the systematic influence that clothes can have on the wearer's feelings, thoughts, and behaviors through their symbolic meaning. It has attracted considerable academic and nonacademic interest, with the 2012 article that coined the phrase cited more than 600 times and covered in more than 160 news outlets. However, a recent high-powered replication failed to replicate one of the original effects. To determine whether the larger body of research on enclothed cognition possesses evidential value and replicable effects, we performed z-curve and meta-analyses using 105 effects from 40 studies across 24 articles (N = 3,789). Underscoring the marked improvement of psychological research practices in the mid-2010s, our results raise concerns about the replicability of early enclothed cognition studies but affirm the evidential value for effects published after 2015. These later studies support the core principle of enclothed cognition-what we wear influences how we think, feel, and act.

Food sharing with friends and acquaintances: A study in preschool boys and girls

Introduction

The current study examined whether preschoolers in a (semi-)natural situation shared more food with friends or acquaintances, and whether this was different between boys and girls, older and younger children, and for preferred and non- preferred food. In order to do so, we replicated and extended the classical work of Birch and Billman in a Dutch sample.

Methods

Participants included 91 children aged between 3 to 6 years (52.7% boys, 93.4% Western European) from a middle- to upper-middle-class neighborhood in the Netherlands.

Results

The results revealed that children shared more non-preferred than preferred food with others. Girls gave more non-preferred food to acquaintances than to friends, whereas boys gave more to friends than to acquaintances. No effect of relationship was found for preferred food. Older children shared more food than younger children. Compared to acquaintances, friends made more active attempts to get food. Moreover, children who were not shared with were just as likely to share food as children who were shared with.

Discussion

Overall, only a small degree of agreement with the original study was found: Some significant findings could not be replicated, and some unconfirmed hypotheses of the original study were supported. The results underscore both the need for replications and studying the effect of social-contextual factors in natural settings.

Multiverse analyses of fear acquisition and extinction retention in posttraumatic stress disorder

Persistent fear is a cardinal feature of posttraumatic stress disorder (PTSD), and deficient fear extinction retention is a proposed illness mechanism and target of exposure-based therapy. However, evidence for deficient fear extinction in PTSD has been mixed using laboratory paradigms, which may relate to underidentified methodological variation across studies. We reviewed the literature to identify parameters that differ across studies of fear extinction retention in PTSD. We then performed Multiverse Analysis in a new sample, to quantify the impact of those methodological parameters on statistical findings. In 25 PTSD patients (15 female) and 36 trauma-exposed non-PTSD controls (TENC) (20 female), we recorded skin conductance response (SCR) during fear acquisition and extinction learning (day 1) and extinction recall (day 2). A first Multiverse Analysis examined the effects of methodological parameters identified by the literature review on comparisons of SCR-based fear extinction retention in PTSD versus TENC. A second Multiverse Analysis examined the effects of those methodological parameters on comparisons of SCR to a danger cue (CS+) versus safety cue (CS-) during fear acquisition. Both the literature review and the Multiverse Analysis yielded inconsistent findings for fear extinction retention in PTSD versus TENC, and most analyses found no statistically significant group difference. By contrast, significantly elevated SCR to CS+ versus CS- was consistently found across all analyses in the literature review and the Multiverse Analysis of new data. We discuss methodological parameters that may most contribute to inconsistent findings of fear extinction retention deficit in PTSD and implications for future clinical research.

The process of replication target selection in psychology: what to consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

Same Question, Different Answers?

Abstract: Psychological scientists increasingly study web data, such as user ratings or social media postings. However, whether research relying on such web data leads to the same conclusions as research based on traditional data is largely unknown. To test this, we (re)analyzed three data sets, thereby comparing web data with laboratory and online survey data. We calculated correlations across these different data sets (Study 1) and investigated identical, illustrative research questions in each data set (Studies 2–4). Our results suggest that web and traditional data are not fundamentally different and usually lead to similar conclusions, but also that it is important to consider differences between data types such as populations and research settings. Web data can be a valuable tool for psychologists when accounting for such differences, as it allows for testing established research findings in new contexts, complementing them with insights from novel data sources.

A data multiverse analysis investigating non-model based SCR quantification approaches

Electrodermal signals are commonly used outcome measures in research on arousal, emotion, and habituation. Recently, we reported on heterogeneity in skin conductance response quantification approaches and its impact on replicability. Here we provide complementary work focusing on within-approach heterogeneity of specifications for skin conductance response quantification. We focus on heterogeneity within the baseline-correction approach (BLC) which appeared as particularly heterogeneous-for instance with respect to the pre-CS baseline window duration, the start, and end of the peak detection window. We systematically scrutinize the robustness of results when applying different BLC approach specifications to one representative pre-existing data set (N = 118) in a (partly) pre-registered study. We report high agreement between different BLC approaches for US and CS+ trials, but moderate to poor agreement for CS- trials. Furthermore, a specification curve of the main effect of CS discrimination during fear acquisition training from all potential and reasonable combinations of specifications (N = 150) and a prototypical trough-to-peak (TTP) approach indicates that resulting effect sizes are largely comparable. A second specification curve (N = 605 specific combinations) highlights a strong impact of different transformation types. Crucially, however, we show that BLC approaches often misclassify the peak value-particularly for CS- trials, leading to stimulus-specific biases and challenges for post-processing and replicability of CS discrimination across studies applying different approaches. Lastly, we investigate how negative skin conductance values in BLC, appearing most frequently for CS- (CS- > CS+ > US), correspond to values in TTP quantification. We discuss the results considering prospects and challenges of the multiverse approach and future directions.

In COVID-19 Health Messaging, Loss Framing Increases Anxiety with Little-to-No Concomitant Benefits: Experimental Evidence from 84 Countries

The COVID-19 pandemic (and its aftermath) highlights a critical need to communicate health information effectively to the global public. Given that subtle differences in information framing can have meaningful effects on behavior, behavioral science research highlights a pressing question: Is it more effective to frame COVID-19 health messages in terms of potential losses (e.g., “If you do not practice these steps, you can endanger yourself and others”) or potential gains (e.g., “If you practice these steps, you can protect yourself and others”)? Collecting data in 48 languages from 15,929 participants in 84 countries, we experimentally tested the effects of message framing on COVID-19-related judgments, intentions, and feelings. Loss- (vs. gain-) framed messages increased self-reported anxiety among participants cross-nationally with little-to-no impact on policy attitudes, behavioral intentions, or information seeking relevant to pandemic risks. These results were consistent across 84 countries, three variations of the message framing wording, and 560 data processing and analytic choices. Thus, results provide an empirical answer to a global communication question and highlight the emotional toll of loss-framed messages. Critically, this work demonstrates the importance of considering unintended affective consequences when evaluating nudge-style interventions.

Examining the generalizability of research findings from archival data

The extent to which results from complex datasets generalize across contexts is critically important to numerous scientific fields as well as to practitioners who rely on such analyses to guide important strategic decisions. Our initiative systematically investigated whether findings from the field of strategic management would emerge in new time periods and new geographies. Original findings that were statistically reliable in the first place were typically obtained again in novel tests, suggesting surprisingly little sensitivity to context. For some social scientific areas of inquiry, results from a specific time and place can be a meaningful guide as to what will be observed more generally.

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Navigating the manyverse of skin conductance response quantification approaches - A direct comparison of trough-to-peak, baseline correction, and model-based approaches in Ledalab and PsPM

Raw data are typically required to be processed to be ready for statistical analyses, and processing pipelines are often characterized by substantial heterogeneity. Here, we applied seven different approaches (trough-to-peak scoring by two different raters, script-based baseline correction, Ledalab as well as four different models implemented in the software PsPM) to two fear conditioning data sets. Selection of the approaches included was guided by a systematic literature search by using fear conditioning research as a case example. Our approach can be viewed as a set of robustness analyses (i.e., same data subjected to different processing pipelines) aiming to investigate if and to what extent these different quantification approaches yield comparable results given the same data. To our knowledge, no formal framework for the evaluation of robustness analyses exists to date, but we may borrow some criteria from a framework suggested for the evaluation of "replicability" in general. Our results from seven different SCR quantification approaches applied to two data sets with different paradigms suggest that there may be no single approach that consistently yields larger effect sizes and could be universally considered "best." Yet, at least some of the approaches employed show consistent effect sizes within each data set indicating comparability. Finally, we highlight substantial heterogeneity also within most quantification approaches and discuss implications and potential remedies.

Accumulating Knowledge in the Organizational Sciences

In some fields, research findings are rigorously curated in a common language and made available to enable future use and large-scale, robust insights. Organizational researchers have begun such efforts [e.g., metaBUS ( http://metabus.org/ )] but are far from the efficient, comprehensive curation seen in areas such as cognitive neuroscience or genetics. This review provides a sample of insights from research curation efforts in organizational research, psychology, and beyond—insights not possible by even large-scale, substantive meta-analyses. Efforts are classified as either science-of-science research or large-scale, substantive research. The various methods used for information extraction (e.g., from PDF files) and classification (e.g., using consensus ontologies) is reviewed. The review concludes with a series of recommendations for developing and leveraging the available corpus of organizational research to speed scientific progress.

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017)

Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.

Both better and worse than others depending on difficulty: Replication and extensions of Kruger’s (1999) above and below average effects

Above-and-below-average effects are well-known phenomena that arise when comparing oneself to others. Kruger (1999) found that people rate themselves as above average for easy abilities and below average for difficult abilities. We conducted a successful pre-registered replication of Kruger’s (1999) Study 1, the first demonstration of the core phenomenon (N = 756, US MTurk workers). Extending the replication to also include a between-subject design, we added two conditions manipulating easy and difficult interpretations of the original ability domains, and with an additional dependent variable measuring perceived difficulty. We observed an above-average-effect in the easy extension and below-average-effect in the difficult extension, compared to the neutral replication condition. Both extension conditions were perceived as less ambiguous than the original neutral condition. Overall, we conclude strong empirical support for Kruger’s above-and-below-average effects, with boundary conditions laid out in the extensions expanding both generalizability and robustness of the phenomenon.

Evidence for goal- and mixed evidence for false belief-based action prediction in two- to four-year-old children: A large-scale longitudinal anticipatory looking replication study

Unsuccessful replication attempts of paradigms assessing children's implicit tracking of false beliefs have instigated the debate on whether or not children have an implicit understanding of false beliefs before the age of four. A novel multi-trial anticipatory looking false belief paradigm yielded evidence of implicit false belief reasoning in three- to four-year-old children using a combined score of two false belief conditions (Grosse Wiesmann, C., Friederici, A. D., Singer, T., & Steinbeis, N. [2017]. Developmental Science, 20(5), e12445). The present study is a large-scale replication attempt of this paradigm. The task was administered three times to the same sample of N = 185 children at two, three, and four years of age. Using the original stimuli, we did not replicate the original finding of above-chance belief-congruent looking in a combined score of two false belief conditions in either of the three age groups. Interestingly, the overall pattern of results was comparable to the original study. Post-hoc analyses revealed, however, that children performed above chance in one false belief condition (FB1) and below chance in the other false belief condition (FB2), thus yielding mixed evidence of children's false belief-based action predictions. Similar to the original study, participants' performance did not change with age and was not related to children's general language skills. This study demonstrates the importance of large-scaled replications and adds to the growing number of research questioning the validity and reliability of anticipatory looking false belief paradigms as a robust measure of children's implicit tracking of beliefs. This article is protected by copyright. All rights reserved.

Risky Therefore Not Beneficial: Replication and Extension of Finucane et al.’s (2000) Affect Heuristic Experiment

Risks and benefits are negatively related in people’s minds. Finucane et al. causally demonstrated that increasing risks of a hazard leads people to judge its benefits as lower. Vice versa, increasing benefits leads people to judge its risks as lower (original: r = −.74 [−0.92, −0.30]). This finding is consistent with an affective explanation, and the negative relationship is often presented as evidence for an affect heuristic. In two well-powered studies, using a more stringent analytic strategy, we replicated the original finding. We observed a strong negative relationship between judgments of risks and benefits across three technologies, although we do find that there was no change in risks when highlighting low benefits. We note that risks seem to be more responsive to manipulation (as opposed to benefits) and find evidence that the negative relationship can depend on incidental mood. We provided materials, data sets, and analyses on https://osf.io/sufjn/?view_only=6f8f5dc6ff524149a4ed5c6de9296ae8 .

PsychOpen CAMA: Publication of community-augmented meta-analyses in psychology

To enable optimal decision‐making based on the best evidence available, open syntheses are called for. To make data accessible and comprehensible even for decision‐makers without proficient knowledge in meta‐analysis, a graphical user interface (GUI) provides flexible data visualizations including interpretation aids. Moreover, due to a growing number of research findings, efficient and easy updating of meta‐analyses is crucial to prevent waste in research. One label for a concept to meet these needs is community‐augmented meta‐analysis (CAMA). The research community at the one hand feeds the data repository of a CAMA with new data and on the other hand benefits from easy access to data and meta‐analyses on a GUI. PsychOpen CAMA has been released recently to serve the psychological research community as a whole by covering a broad scope of potential research domains. PsychOpen CAMA relies on a web application with an OpenCPU server for the R calculations. To achieve interoperability of different datasets with the analysis functions used in PsychOpen CAMA, a template for meta‐analytic data and machine‐readable metadata are used. In the future, the automation of workflows, flexibility of analysis options, and the scope of the platform will be further developed by making use of synergies with other resources and tools at ZPID. The article provides an overview on the rationale for the necessity of open syntheses and the CAMA approach, as well as a presentation of the architecture, user interface, functionalities and future challenges of PsychOpen CAMA.

Agency and self-other asymmetries in perceived bias and shortcomings: Replications of the Bias Blind Spot and link to free will beliefs

Bias Blind Spot (BBS) is the phenomenon that people tend to perceive themselves as less susceptible to biases than others. In three pre-registered experiments (overall N = 969), we replicated two experiments of the first demonstration of the phenomenon by Pronin et al. (2002). We found support of the BBS hypotheses, with effects in line with findings in the original study: Participants rated themselves as less susceptible to biases than others (d = –1.00 [–1.33, –0.67]). Deviating from the original, we found an unexpected effect that participants rated themselves as having fewer shortcomings (d = –0.34 [–0.46, –0.23]), though there was support for the target’s main premise that BBS was stronger for biases than for shortcomings (d = –0.43 [–0.56, –0.29]). Extending the replications, we found that beliefs in own free will were positively associated with BBS (r ∼ 0.17–0.22) and that beliefs in both own and general free will were positively associated with self-other asymmetry related to personal shortcomings (r ∼ 0.16–0.24). Materials, datasets, and code are available on https://osf.io/3df5s/.

Replicability, Robustness, and Reproducibility in Psychological Science

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies

Preregistration is a method to increase research transparency by documenting research decisions on a public, third-party repository prior to any influence by data. It is becoming increasingly popular in all subfields of psychology and beyond. Adherence to the preregistration plan may not always be feasible and even is not necessarily desirable, but without disclosure of deviations, readers who do not carefully consult the preregistration plan might get the incorrect impression that the study was exactly conducted and reported as planned. In this paper, we have investigated adherence and disclosure of deviations for all articles published with the Preregistered badge in Psychological Science between February 2015 and November 2017 and shared our findings with the corresponding authors for feedback. Two out of 27 preregistered studies contained no deviations from the preregistration plan. In one study, all deviations were disclosed. Nine studies disclosed none of the deviations. We mainly observed (un)disclosed deviations from the plan regarding the reported sample size, exclusion criteria and statistical analysis. This closer look at preregistrations of the first generation reveals possible hurdles for reporting preregistered studies and provides input for future reporting guidelines. We discuss the results and possible explanations, and provide recommendations for preregistered research.

Understanding Replication in a Way That Is True to Science

In this article, I critically examine a number of widely held beliefs about the nature of replication and its place in science, with particular reference to psychology. In doing so, I present a number of underappreciated understandings of the nature of science more generally. I contend that some contributors to the replication debates overstate the importance of replication in science and mischaracterize the relationship between direct and conceptual replication. I also claim that there has been a failure to appreciate sufficiently the variety of legitimate replication practices that scientists engage in. In this regard, I highlight the tendency to pay insufficient attention to methodological triangulation as an important strategy for justifying empirical claims. I argue, further, that the replication debates tend to overstate the closeness of the relationship between replication and theory construction. Some features of this relationship are spelt out with reference to the hypothetico-deductive and the abductive accounts of scientific method. Additionally, an evaluation of the status of replication in different characterizations of scientific progress is undertaken. I maintain that viewing replication as just one element of the wide array of scientific endeavors leads to the conclusion that it is not as prominent in science as is often claimed.

"What We Have Here, Is a Failure to [Replicate]": Ways to Solve a Replication Crisis in Implementation Science

Adapting the classic line from the 1967 film Cool Hand Luke, the title is meant to convey that implementation science (IS), like other fields, has not been embracing replication studies, which is a key component to the open science movement. The purpose of this article is to review what is known about replication of implementation trials and identify the gaps and next steps to continue increasing the transparency, openness, and replicability of implementation research. After presenting an overview of study replication and how it is a key component of open science, the article will examine how replication of implementation studies has (or more accurately has not) been approached in IS. As will be discussed, replication in IS shares some challenges with studies that attempt to replicate interventions, but also presents unique challenges. This article discusses different types of replications (e.g., direct vs. conceptual) and how they can benefit the field of IS. The article then presents a specific example of an implementation strategy called Getting To Outcomes© to describe how to design a replication study and interpret the results. The article ends with multiple options implementation scientists could consider to improve the likelihood and quality of replication studies. The discussion also envisions how implementation science can enable researchers and practitioners to work together in real-world contexts to encourage wide replication of implementation studies and advance the goal of improving public health.

Context Dependency as a Predictor of Replicability

We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

Design-Based Approaches to Causal Replication Studies

Recent interest in promoting replication efforts assumes that there is well-established methodological guidance for designing and implementing these studies. However, no such consensus exists in the methodology literature. This article addresses these challenges by describing design-based approaches for planning systematic replication studies. Our general approach is derived from the Causal Replication Framework (CRF), which formalizes the assumptions under which replication success can be expected. The assumptions may be understood broadly as replication design requirements and individual study design requirements. Replication failure occurs when one or more CRF assumptions are violated. In design-based approaches to replication, CRF assumptions are systematically tested to evaluate the replicability of effects, as well as to identify sources of effect variation when replication failure is observed. The paper describes research designs for replication and demonstrates how multiple designs may be combined in systematic replication efforts, as well as how diagnostic measures may be used to assess the extent to which CRF assumptions are met in field settings.

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

Computational reproducibility is a corner stone for sound and credible research. Especially in complex statistical analyses-such as the analysis of longitudinal data-reproducing results is far from simple, especially if no source code is available. In this work we aimed to reproduce analyses of longitudinal data of 11 articles published in PLOS ONE. Inclusion criteria were the availability of data and author consent. We investigated the types of methods and software used and whether we were able to reproduce the data analysis using open source software. Most articles provided overview tables and simple visualisations. Generalised Estimating Equations (GEEs) were the most popular statistical models among the selected articles. Only one article used open source software and only one published part of the analysis code. Replication was difficult in most cases and required reverse engineering of results or contacting the authors. For three articles we were not able to reproduce the results, for another two only parts of them. For all but two articles we had to contact the authors to be able to reproduce the results. Our main learning is that reproducing papers is difficult if no code is supplied and leads to a high burden for those conducting the reproductions. Open data policies in journals are good, but to truly boost reproducibility we suggest adding open code policies.

No links between genetic variation and developing theory of mind: A preregistered replication attempt of candidate gene studies

Genetic variability is being discussed as a source of inter-individual differences in Theory of Mind development. Previous studies documented an association between variations in DRD4 VNTR 48 bp, OXTR rs53576, COMT rs4680, and Theory of Mind task performance. As empirical evidence on these associations is sparse, we conducted a preregistered replication attempt of a study reporting a link between DRD4 VNTR 48 bp and false belief understanding in 50-month-old children [Lackner, C., Sabbagh, M. A., Hallinan, E., Liu, X., & Holden, J. J. (2012). Developmental Science, 15(2), 272-280.]. Additionally, we attempted a replication of studies on the role of OXTR rs53576 and COMT rs4680 in Theory of Mind. In both replication attempts, we did not find any evidence for associations between the sampled genetic markers and Theory of Mind ability in a series of analyses. Extending the replication attempt of Lackner et al., we employed longitudinal data from several tasks and measurement points, which allowed us to run follow-up robustness checks with more reliable scores. These extensive analyses corroborated our null finding. This comprehensive non-replication is important to balance current research on genetic markers of Theory of Mind. In a combined evaluation of our own and previous studies, we point to substantial methodological issues that research on the genetic basis of Theory of Mind development faces. We conclude that these limitations currently prevent firm conclusions on genetic influences on Theory of Mind development.

Do We Report the Information That Is Necessary to Give Psychology Away? A Scoping Review of the Psychological Intervention Literature 2000-2018

Psychologists are spending a considerable amount of time researching and developing interventions in hopes that their efforts can help to tackle some of society's pressing problems. Unfortunately, those hopes are often not realized-many interventions are developed and reported in journals but do not make their way into the broader world they were designed to change. One potential reason for this is that there may be a gap between the information reported in articles and the information others, such as practitioners, need to implement the findings. We explored this possibility in the current article. We conducted a scoping review to assess the extent to which the information needed for implementation is reported in psychological intervention articles. Results suggest psychological intervention articles report, at most, 64% of the information needed to implement interventions. We discuss the implications of this for both psychological theories and applying them in the world.

Risky choice framing with various problem descriptions: A replication and extension study

In “Risky choice framing: Task versions and a comparison of prospect theory and fuzzy-trace theory”, Kühberger and Tanner (2010) examined the impacts of removing stated zero/non-zero complements of risky options on the gain/loss framing effect. They also tested two rival theoretical explanations for this effect: prospect theory and fuzzy-trace theory. The present study aimed to examine the reliability and robustness of the evidence provided by Kühberger and Tanner by precise replication in Study 1. The original findings were reported for conditions in which the probability of the risky option was fixed, and the expected value of the two alternatives was approximately equivalent. The present study also aimed to examine the generality of their findings under additional conditions in which large, medium and small probabilities of the risky option were assigned, and the expected value of the certain or risky options differed. The main findings of Kühberger and Tanner (2010) were successfully replicated and confirmed under the original and additional conditions. The implications of these findings are discussed.

Testing a computational model of subjective well-being: a preregistered replication of Rutledge et al. (2014)

Subjective well-being changes over time. While the causes of these changes have been investigated extensively, few attempts have been made to capture these changes through computational modelling. One notable exception is the study by Rutledge et al. [Rutledge, R. B., Skandali, N., Dayan, P., & Dolan, R. J. (2014). A computational and neural model of momentary subjective well-being. Proceedings of the National Academy of Sciences, 111(33), 12252-12257. https://doi.org/10.1073/pnas.1407535111], in which a model that captures momentary changes in subjective well-being was proposed. The model incorporates how an individual processes rewards and punishments in a decision context. Using this model, the authors were able to successfully explain fluctuations in subjective well-being observed in a gambling paradigm. Although Rutledge et al. reported an in-paper replication, a successful independent replication would further increase the credibility of their results. In this paper, we report a preregistered close replication of the behavioural experiment and analyses by Rutledge et al. The results of Rutledge et al. were mostly confirmed, providing further evidence for the role of rewards and punishments in subjective well-being fluctuations. Additionally, the association between personality traits and the way people process rewards and punishments was examined. No evidence for such associations was found, leaving this an open question for future research.

Honest signaling in academic publishing

Academic journals provide a key quality-control mechanism in science. Yet, information asymmetries and conflicts of interests incentivize scientists to deceive journals about the quality of their research. How can honesty be ensured, despite incentives for deception? Here, we address this question by applying the theory of honest signaling to the publication process. Our models demonstrate that several mechanisms can ensure honest journal submission, including differential benefits, differential costs, and costs to resubmitting rejected papers. Without submission costs, scientists benefit from submitting all papers to high-ranking journals, unless papers can only be submitted a limited number of times. Counterintuitively, our analysis implies that inefficiencies in academic publishing (e.g., arbitrary formatting requirements, long review times) can serve a function by disincentivizing scientists from submitting low-quality work to high-ranking journals. Our models provide simple, powerful tools for understanding how to promote honest paper submission in academic publishing.

Accentuation and compatibility: Replication and extensions of Shafir (1993) to rethink choosing versus rejecting paradigms

We conducted a replication of Shafir (1993) who showed that people are inconsistent in their preferences when faced with choosing versus rejecting decision-making scenarios. The effect was demonstrated using an enrichment paradigm, asking subjects to choose between enriched and impoverished alternatives, with enriched alternatives having more positive and negative features than the impoverished alternative. Using eight different decision scenarios, Shafir found support for a compatibility principle: subjects chose and rejected enriched alternatives in choose and reject decision scenarios (d = 0.32 [0.23,0.40]), respectively, and indicated greater preference for the enriched alternative in the choice task than in the rejection task (d = 0.38 [0.29,0.46]). In a preregistered very close replication of the original study (N = 1026), we found no consistent support for the hypotheses across the eight problems: two had similar effects, two had opposite effects, and four showed no effects (overall d = −0.01 [−0.06,0.03]). Seeking alternative explanations, we tested an extension, and found support for the accentuation hypothesis.

Community-Augmented Meta-Analyses (CAMAs) in Psychology

Abstract. The limits of static snapshot meta-analyses and the relevance of reproducibility and data accessibility for cumulative meta-analytic research are outlined. A publication format to meet these requirements is presented: Community-augmented meta-analyses (CAMA). We give an overview of existing systems implementing this approach and compare these in terms of scope, technical implementation, data collection and augmentation, data curation, tools available for analysis, and methodological flexibility.

The "replication crisis" in the public eye: Germans' awareness and perceptions of the (ir)reproducibility of scientific research

Several meta-analytical attempts to reproduce results of empirical research have failed in recent years, prompting scholars and news media to diagnose a "replication crisis" and voice concerns about science losing public credibility. Others, in contrast, hoped replication efforts could improve public confidence in science. Yet nationally representative evidence backing these concerns or hopes is scarce. We provide such evidence, conducting a secondary analysis of the German "Science Barometer" ("Wissenschaftsbarometer") survey. We find that most Germans are not aware of the "replication crisis." In addition, most interpret replication efforts as indicative of scientific quality control and science's self-correcting nature. However, supporters of the populist right-wing party AfD tend to believe that the "crisis" shows one cannot trust science, perhaps using it as an argument to discredit science. But for the majority of Germans, hopes about reputational benefits of the "replication crisis" for science seem more justified than concerns about detrimental effects.

Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study

For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one 'major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility.

Revisiting potential associations between brain morphology, fear acquisition and extinction through new data and a literature review

Inter-individual differences in defensive responding are widely established but their morphological correlates in humans have not been investigated exhaustively. Previous studies reported associations with cortical thickness of the dorsal anterior cingulate cortex, insula and medial orbitofrontal cortex as well as amygdala volume in fear conditioning studies. However, these associations are partly inconsistent and often derived from small samples. The current study aimed to replicate previously reported associations between physiological and subjective measures of fear acquisition and extinction and brain morphology. Structural magnetic resonance imaging was performed on 107 healthy adults who completed a differential cued fear conditioning paradigm with 24 h delayed extinction while skin conductance response (SCR) and fear ratings were recorded. Cortical thickness and subcortical volume were obtained using the software Freesurfer. Results obtained by traditional null hypothesis significance testing and Bayesians statistics do not support structural brain-behavior relationships: Neither differential SCR nor fear ratings during fear acquisition or extinction training could be predicted by cortical thickness or subcortical volume in regions previously reported. In summary, the current pre-registered study does not corroborate associations between brain morphology and inter-individual differences in defensive responding but differences in experimental design and analyses approaches compared to previous work should be acknowledged.