The replication crisis has led to positive structural, procedural, and community changes

The hidden complexity of the simple world of basic experimental psychology: the principal and practical limits of gaining psychological knowledge using the experimental method

Basic experimental research in psychology is based on the assumption that law-like behavior can be observed if the complexity of the human psyche is reduced by the creation of experimental settings in which simple psychical phenomena occur which reflect the effect of an isolated psychological mechanism. However, we show that this assumption does not hold for many phenomena studied in basic experimental psychology because even phenomena that are regarded as simple and fully controllable often fluctuate unpredictably as a function of unintentionally chosen details of the experimental setting. The reason is that in a complex system like the human psyche, even minimal, and from the perspective of the investigated research question irrelevant, differences in the experimental setting can build up to large unsystematic effects. Law-like behavior in experiments could only occur if truly low-level mechanisms were studied in a truly isolated way. However, this is often not the case in current experimental research. One problem is that often fuzzy theoretical terms are used which only give the impression that low-level mechanisms are being investigated, although in reality the complexity of the human psyche is unintentionally brought on board. Another problem is that, unlike in the natural sciences, the mechanisms of the human psyche can only be isolated from each other to a limited extent because the human psyche always reacts as a whole system. If such problems could be overcome, meaningful knowledge could be gained through experimental psychological research. However, the knowledge gained is very limited in terms of its explanatory power for human behavior, as it is only helpful for understanding a very specific aspect of behavior, namely the mechanistic functioning of isolated low-level mechanisms. When it comes to understanding motivated behavior in real life, knowledge about the non-mechanistic functioning of the higher levels of the human psyche is necessary, but this knowledge cannot be gained through the experimental method.

Reproducibility and replicability in research: What 452 professors think in Universities across the USA and India

In the past decade, open science and science of science communities have initiated innovative efforts to address concerns about the reproducibility and replicability of published scientific research. In some respects, these efforts have been successful, yet there are still many pockets of researchers with little to no familiarity with these concerns, subsequent responses, or best practices for engaging in reproducible, replicable, and reliable scholarship. In this study, we surveyed 452 professors from universities across the USA and India to understand perspectives on scientific processes and identify key points for intervention. Our findings reveal both national and disciplinary gaps in attention to reproducibility and transparency in science, aggravated by incentive misalignment and resource constraints. We suggest that solutions addressing scientific integrity should be culturally-centered, where definitions of culture should include both regional and domain-specific elements. This study examines research cultures in India and the USA across a diverse range of social science and engineering disciplines. The universities included in the study were carefully selected to represent various regions of each country and reflect institutions across different ranking levels, ensuring a broad and representative sample. While the findings provide valuable insights into the research environments of India and the USA, their applicability is limited to these two countries and respective disciplines. The survey relies on self-reported data, which can be subject to biases, e.g., social desirability or recall bias. Future research will expand the scope to include additional countries, allowing for a more comprehensive comparison of global research cultures. Additionally, we aim to investigate how regional, institutional, and disciplinary factors influence research practices and collaboration across borders, providing a deeper understanding of international academic environments.

International initiatives to enhance awareness and uptake of open research in psychology: a systematic mapping review

Concerns about the replicability, reproducibility and transparency of research have ushered in a set of practices and behaviours under the umbrella of 'open research'. To this end, many new initiatives have been developed that represent procedural (i.e. behaviours and sets of commonly used practices in the research process), structural (new norms, rules, infrastructure and incentives), and community-based change (working groups, networks). The objectives of this research were to identify and outline international initiatives that enhance awareness and uptake of open research practices in the discipline of psychology. A systematic mapping review was conducted in three stages: (i) a Web search to identify open research initiatives in psychology; (ii) a literature search to identify related articles; and (iii) a hand search of grey literature. Eligible initiatives were then coded into an overarching theme of procedural, structural or community-based change. A total of 187 initiatives were identified; 30 were procedural (e.g. toolkits, resources, software), 70 structural (e.g. policies, strategies, frameworks) and 87 community-based (e.g. working groups, networks). This review highlights that open research is progressing at pace through various initiatives that share a common goal to reform research culture. We hope that this review promotes their further adoption and facilitates coordinated efforts between individuals, organizations, institutions, publishers and funders.

What we mean when we say semantic: Toward a multidisciplinary semantic glossary

Tulving characterized semantic memory as a vast repository of meaning that underlies language and many other cognitive processes. This perspective on lexical and conceptual knowledge galvanized a new era of research undertaken by numerous fields, each with their own idiosyncratic methods and terminology. For example, "concept" has different meanings in philosophy, linguistics, and psychology. As such, many fundamental constructs used to delineate semantic theories remain underspecified and/or opaque. Weak construct specificity is among the leading causes of the replication crisis now facing psychology and related fields. Term ambiguity hinders cross-disciplinary communication, falsifiability, and incremental theory-building. Numerous cognitive subdisciplines (e.g., vision, affective neuroscience) have recently addressed these limitations via the development of consensus-based guidelines and definitions. The project to follow represents our effort to produce a multidisciplinary semantic glossary consisting of succinct definitions, background, principled dissenting views, ratings of agreement, and subjective confidence for 17 target constructs (e.g., abstractness, abstraction, concreteness, concept, embodied cognition, event semantics, lexical-semantic, modality, representation, semantic control, semantic feature, simulation, semantic distance, semantic dimension). We discuss potential benefits and pitfalls (e.g., implicit bias, prescriptiveness) of these efforts to specify a common nomenclature that other researchers might index in specifying their own theoretical perspectives (e.g., They said X, but I mean Y).

A manifesto for a globally diverse, equitable, and inclusive open science

The field of psychology has rapidly transformed its open science practices in recent years. Yet there has been limited progress in integrating principles of diversity, equity and inclusion. In this Perspective, we raise the spectre of Questionable Generalisability Practices and the issue of MASKing (Making Assumptions based on Skewed Knowledge), calling for more responsible practices in generalising study findings and co-authorship to promote global equity in knowledge production. To drive change, researchers must target all four key components of the research process: design, reporting, generalisation, and evaluation. Additionally, macro-level geopolitical factors must be considered to move towards a robust behavioural science that is truly inclusive, representing the voices and experiences of the majority world (i.e., low-and-middle-income countries).

A framework for joint music making: Behavioral findings, neural processes, and computational models

Across different epochs and societies, humans occasionally gather to jointly make music. This universal form of collective behavior is as fascinating as it is fragmentedly understood. As the interest in joint music making (JMM) rapidly grows, we review the state-of-the-art of this emerging science, blending behavioral, neural, and computational contributions. We present a conceptual framework synthesizing research on JMM within four components. The framework is centered upon interpersonal coordination, a crucial requirement for JMM. The other components imply the influence of individuals' (past) experience, (current) social factors, and (future) goals on real-time coordination. Our aim is to promote the development of JMM research by organizing existing work, inspiring new questions, and fostering accessibility for researchers belonging to other research communities.

Building transparency and reproducibility into the practice of pharmacoepidemiology and outcomes research

Real-world evidence (RWE) studies are increasingly used to inform policy and clinical decisions. However, there remain concerns about the credibility and reproducibility of RWE studies. While there is universal agreement on the critical importance of transparent and reproducible science, the building blocks of open science practice that are common across many disciplines have not yet been built into routine workflows for pharmacoepidemiology and outcomes researchers. Observational researchers should highlight the level of transparency of their studies by providing a succinct statement addressing study transparency with the publication of every paper, poster, or presentation that reports on an RWE study. In this paper, we propose a framework for an explicit transparency statement that declares the level of transparency a given RWE study has achieved across 5 key domains: (1) protocol, (2) preregistration, (3) data, (4) code-sharing, and (5) reporting checklists. The transparency statement outlined in the present paper can be used by research teams to proudly display the open science practices that were used to generate evidence designed to inform public health policy and practice. While transparency does not guarantee validity, such a statement signals confidence from the research team in the scientific choices that were made.

Housing and Husbandry Factors Affecting Zebrafish (Danio rerio) Novel Tank Test Responses: A Global Multi-Laboratory Study

The reproducibility crisis in bioscience, characterized by inconsistent study results, impedes our understanding of biological processes and global collaborative studies offer a unique solution. This study is the first global collaboration using the zebrafish (Danio rerio) novel tank test, a behavioral assay for anxiety-like responses. We analyzed data from 20 laboratories worldwide, focusing on housing conditions and experimental setups. Our study included 488 adult zebrafish, tested for 5 min, focusing on a variety of variables. Key findings show females exhibit more anxiety-like behavior than males, underscoring sex as a critical variable. Housing conditions, including higher stocking densities and specific feed types, influenced anxiety levels. Optimal conditions (5 fish/L) and nutritionally rich feeds (e.g., rotifers), mitigated anxiety-like behaviors. Environmental stressors, like noise and transportation, significantly impacted behavior. We recommend standardizing protocols to account for sex differences, optimal stocking densities, nutritionally rich feeds, and minimizing stressors to improve zebrafish behavioral study reliability.

Predictive Utility of the Multi-Process Action Control Framework for Self-Reported and Device-Measured Physical Activity Behavior of Adolescents

Understanding the correlates of physical activity behavior is imperative for informing the development of interventions to address the low rates of physical activity guideline adherence among adolescents living in the United States. This cross-sectional study examined the predictive utility of the Multi-Process Action Control (M-PAC) framework for explaining self-reported and device-measured physical activity behavior among a Hispanic-majority sample of adolescents. A total of 1849 high school students (mean age = 16.0 ± 1.22 SD years; 52.3% women; 87.8% Hispanic) enrolled in one school district in south-central Texas completed a survey including instruments to assess M-PAC framework constructs (instrumental and affective attitudes, perceived capability and opportunity, behavioral regulation, habit, identity) and moderate-to-vigorous physical activity (MVPA) behavior. A subsample (n = 435) wore accelerometers for seven days. The results from robust linear regression models revealed role identity and habit were significant predictors of self-reported MVPA. Role identity was a significant predictor of accelerometer-derived daily MVPA and raw acceleration in the most active hour but not daily raw acceleration. The findings indicated reflexive processes are robust predictors of adolescent physical activity and should be the focus of interventions designed to promote adoption and maintenance of physical activity during this developmental life stage.

The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche

Many suggestions for dealing with the so-called replication crisis in psychology revolve around the idea that better and more complex statistical-mathematical tools or stricter procedures are required in order to obtain reliable findings and prevent cheating or publication biases. While these aspects may play an exacerbating role, we interpret the replication crisis primarily as an epistemological crisis in psychology caused by an inadequate fit between the ontic nature of the psyche and the quantitative approach. On the basis of the philosophers of science Karl Popper, Thomas Kuhn, and Imre Lakatos we suggest that the replication crisis is therefore a symptom of a fundamental problem in psychology, but at the same time it is also an opportunity to advance psychology as a science. In a first step, against the background of Popper's Critical Rationalism, the replication crisis is interpreted as an opportunity to eliminate inaccurate theories from the pool of theories and to correct problematic developments. Continuing this line of thought, in an interpretation along the lines of Thomas Kuhn, the replication crisis might signify a model drift or even model crisis, thus possibly heralding a new paradigm in psychology. The reasons for this are located in the structure of academic psychology on the basis of Lakatos's assumption about how sciences operate. Accordingly, one hard core that lies at the very basis of psychology may be found in the assumption that the human psyche can and is to be understood in quantitative terms. For this to be possible, the ontic structure of the psyche, i.e., its very nature, must also in some way be quantitatively constituted. Hence, the replication crisis suggests that the ontic structure of the psyche in some way (also) contains a non-quantitative dimension that can only be grasped incompletely or fragmentarily using quantitative research methods. Fluctuating and inconsistent results in psychology could therefore also be the expression of a mismatch between the ontic level of the object of investigation and the epistemic level of the investigation.

The dynamics of sociality and glucocorticoids in wild male Assamese macaques

For males of gregarious species, dominance status and the strength of affiliative relationships can have major fitness consequences. Social dynamics also impose costs by affecting glucocorticoids, mediators of homeostasis and indicators of the physiological response to challenges and within-group competition. We investigated the relationships between dominance, social bonds, seasonal challenges, and faecal glucocorticoid metabolite (fGC) measures in wild Assamese macaques (Macaca assamensis) at Phu Khieo Wildlife Sanctuary, Thailand, combining behavioural data with 4129 samples from 62 adult males over 15 years. Our previous work on this population suggested that increased competition during the mating season was associated with elevated fGC levels and that, unusually for male primates, lower rank position correlated with higher fGC levels. With a much larger dataset and dynamic measures of sociality, we re-examined these relationships and additionally tested the potentially fGC-attenuating effect of social support. Contrary to our previous study, yet consistent with the majority of work on male primates, dominance rank had a positive relationship with fGC levels, as high status correlated with elevated glucocorticoid measures. fGC levels were increased at the onset of the mating season. We demonstrated an fGC-reducing effect of supportive relationships in males and showed that dynamics in affiliation can correlate with dynamics in physiological responses. Our results suggest that in a system with intermediate contest potential, high dominance status can impose physiological costs on males that may potentially be moderated by social relationships. We highlight the need to consider the dynamics of sociality and competition that influence hormonal processes.

Transparency and reproducibility in the Adolescent Brain Cognitive Development (ABCD) study

BACKGROUND

Transparency can build trust in the scientific process, but scientific findings can be undermined by poor and obscure data use and reporting practices. The purpose of this work is to report how data from the Adolescent Brain Cognitive Development (ABCD) Study has been used to date, and to provide practical recommendations on how to improve the transparency and reproducibility of findings.

METHODS

Articles published from 2017 to 2023 that used ABCD Study data were reviewed using more than 30 data extraction items to gather information on data use practices. Total frequencies were reported for each extraction item, along with computation of a Level of Completeness (LOC) score that represented overall endorsement of extraction items. Univariate linear regression models were used to examine the correlation between LOC scores and individual extraction items. Post hoc analysis included examination of whether LOC scores were correlated with the logged 2-year journal impact factor.

RESULTS

There were 549 full-length articles included in the main analysis. Analytic scripts were shared in 30 % of full-length articles. The number of participants excluded due to missing data was reported in 60 % of articles, and information on missing data for individual variables (e.g., household income) was provided in 38 % of articles. A table describing the analytic sample was included in 83 % of articles. A race and/or ethnicity variable was included in 78 % of reviewed articles, while its inclusion was justified in only 41 % of these articles. LOC scores were highly correlated with extraction items related to examination of missing data. A bottom 10 % of LOC score was significantly correlated with a lower logged journal impact factor when compared to the top 10 % of LOC scores (β=-0.77, 95 % -1.02, -0.51; p-value < 0.0001).

CONCLUSION

These findings highlight opportunities for improvement in future papers using ABCD Study data to readily adapt analytic practices for better transparency and reproducibility efforts. A list of recommendations is provided to facilitate adherence in future research.

Transparency and Reproducibility in the Adolescent Brain Cognitive Development (ABCD) Study

Background

Transparency can build trust in the scientific process, but scientific findings can be undermined by poor and obscure data use and reporting practices. The purpose of this work is to report how data from the Adolescent Brain Cognitive Development (ABCD) Study has been used to date, and to provide practical recommendations on how to improve the transparency and reproducibility of findings.

Methods

Articles published from 2017 to 2023 that used ABCD Study data were reviewed using more than 30 data extraction items to gather information on data use practices. Total frequencies were reported for each extraction item, along with computation of a Level of Completeness (LOC) score that represented overall endorsement of extraction items. Univariate linear regression models were used to examine the correlation between LOC scores and individual extraction items. Post hoc analysis included examination of whether LOC scores were correlated with the logged 2-year journal impact factor.

Results

There were 549 full-length articles included in the main analysis. Analytic scripts were shared in 30% of full-length articles. The number of participants excluded due to missing data was reported in 60% of articles, and information on missing data for individual variables (e.g., household income) was provided in 38% of articles. A table describing the analytic sample was included in 83% of articles. A race and/or ethnicity variable was included in 78% of reviewed articles, while its inclusion was justified in only 41% of these articles. LOC scores were highly correlated with extraction items related to examination of missing data. A bottom 10% of LOC score was significantly correlated with a lower logged journal impact factor when compared to the top 10% of LOC scores (β=-0.77, 95% -1.02, -0.51; p-value < 0.0001).

Conclusion

These findings highlight opportunities for improvement in future papers using ABCD Study data to readily adapt analytic practices for better transparency and reproducibility efforts. A list of recommendations is provided to facilitate adherence in future research.

Hyper-ambition and the Replication Crisis: Why Measures to Promote Research Integrity can Falter

This paper introduces the concept of 'hyper-ambition' in academia as a contributing factor to what has been termed a 'replication crisis' across some sciences. The replication crisis is an umbrella term that covers a range of 'questionable research practices', from sloppy reporting to fraud. There are already many proposals to address questionable research practices, some of which focus on the values, norms, and motivations of researchers and institutes, and suggest measures to promote research integrity. Yet it is not easy to promote integrity in hyper-competitive academic environments that value high levels of ambition. I argue that in such contexts, it is as likely that a kind of hyper-ambition is fostered that (inadvertently or otherwise) prioritises individual success above all, including to the detriment of scientific quality. In addition, efforts to promote values like integrity falter because they rely on sufficient uniformity in motivations or tendencies. Codes and guidance promoting integrity are, however, likely to influence those for whom such values are not optional, while others simply find ways around them. To demonstrate this I offer a thought experiment in which we consider the imaginary working situations of two ordinary academics. I conclude that tackling questionable research practices in the light of the replication crisis requires robust 'top down' measures that expect and accommodate a broader range of academic values, motivations, and tendencies, while challenging those that help to promote hyper-ambition.

On hidden factors and design-associated errors that may lead to data misinterpretation: An example from preclinical research on the potential seasonality of neonatal seizures

Unintentional misinterpretation of research in published biomedical reports that is not based on statistical flaws is often underrecognized, despite its possible impact on science, clinical practice, and public health. Important causes of such misinterpretation of scientific data, resulting in either false positive or false negative conclusions, include design-associated errors and hidden (or latent) variables that are not easily recognized during data analysis. Furthermore, cognitive biases, such as the inclination to seek patterns in data whether they exist or not, may lead to misinterpretation of data. Here, we give an example of these problems from hypothesis-driven research on the potential seasonality of neonatal seizures in a rat model of birth asphyxia. This commentary aims to raise awareness among the general scientific audience about the issues related to the presence of unintentional misinterpretation in published reports.

Big team science initiatives: A catalyst for trustworthy advancements in IO psychology

Keener et al. (2023) raise concerns about the trustworthiness of Industrial/Organizational (IO) Psychology research and related fields due to the low reproducibility and replicability of research findings. The authors provide various solutions to resolve this crisis, such as improving training, realigning incentives, and adopting open science practices. Our commentary elaborates on one solution to which they briefly allude: Big Team Science Initiatives (BTSIs). BTSIs allow scholars to address the trustworthiness of our science by facilitating large sample theory testing, sharing and allocating resources, and selecting appropriate research strategies, all of which support the reproducibility and replication of research. Further, we propose that BTSIs may facilitate researcher training, encourage data sharing and materials, and realign incentives in our field. We discuss how BTSIs could be implemented in IO psychology and related fields, identifying and drawing upon similar BTSIs in related disciplines. Thus, our commentary is an extension of the focal article, encouraging scholars to collaboratively address the "crisis of confidence" facing our field using a big team science approach.

Reducing intervention- and research-induced inequalities to tackle the digital divide in health promotion

Social inequalities are an important contributor to the global burden of disease within and between countries. Using digital technology in health promotion and healthcare is seen by some as a potential lever to reduce these inequalities; however, research suggests that digital technology risks re-enacting or evening widening disparities. Most research on this digital health divide focuses on a small number of social inequality indicators and stems from Western, educated, industrialized, rich, and democratic (WEIRD) countries. There is a need for systematic, international, and interdisciplinary contextualized research on the impact of social inequality indicators in digital health as well as the underlying mechanisms of this digital divide across the globe to reduce health disparities. In June 2023, eighteen multi-disciplinary researchers representing thirteen countries from six continents came together to discuss current issues in the field of digital health promotion and healthcare contributing to the digital divide. Ways that current practices in research contribute to the digital health divide were explored, including intervention development, testing, and implementation. Based on the dialogue, we provide suggestions for overcoming barriers and improving practices across disciplines, countries, and sectors. The research community must actively advocate for system-level changes regarding policy and research to reduce the digital divide and so improve digital health for all.