Replication of null results: Absence of evidence or evidence of absence?

In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a 'replication success.' Here, we discuss the logical problems with this approach: Non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect and 'replication success' can virtually always be achieved if the sample sizes are small enough. In addition, the relevant error rates are not controlled. We show how methods, such as equivalence testing and Bayes factors, can be used to adequately quantify the evidence for the absence of an effect and how they can be applied in the replication setting. Using data from the Reproducibility Project: Cancer Biology, the Experimental Philosophy Replicability Project, and the Reproducibility Project: Psychology we illustrate that many original and replication studies with 'null results' are in fact inconclusive. We conclude that it is important to also replicate studies with statistically non-significant results, but that they should be designed, analyzed, and interpreted appropriately.

Evidence for Infant-directed Speech Preference Is Consistent Across Large-scale, Multi-site Replication and Meta-analysis

There is substantial evidence that infants prefer infant-directed speech (IDS) to adult-directed speech (ADS). The strongest evidence for this claim has come from two large-scale investigations: i) a community-augmented meta-analysis of published behavioral studies and ii) a large-scale multi-lab replication study. In this paper, we aim to improve our understanding of the IDS preference and its boundary conditions by combining and comparing these two data sources across key population and design characteristics of the underlying studies. Our analyses reveal that both the meta-analysis and multi-lab replication show moderate effect sizes (d ≈ 0.35 for each estimate) and that both of these effects persist when relevant study-level moderators are added to the models (i.e., experimental methods, infant ages, and native languages). However, while the overall effect size estimates were similar, the two sources diverged in the effects of key moderators: both infant age and experimental method predicted IDS preference in the multi-lab replication study, but showed no effect in the meta-analysis. These results demonstrate that the IDS preference generalizes across a variety of experimental conditions and sampling characteristics, while simultaneously identifying key differences in the empirical picture offered by each source individually and pinpointing areas where substantial uncertainty remains about the influence of theoretically central moderators on IDS preference. Overall, our results show how meta-analyses and multi-lab replications can be used in tandem to understand the robustness and generalizability of developmental phenomena.

Exploring the multiverse of analysis options for the alcohol Stroop

The alcohol Stroop is a widely used task in addiction science to measure the theoretical concept of attentional bias (a selective attention to alcohol-related cues in the environment), which is thought to be associated with clinical outcomes (craving and consumption). However, recent research suggests findings from this task can be equivocal. This may be because the task has many different potential analysis pipelines, which increase researcher degrees of freedom when analysing data and reporting results. These analysis pipelines largely come from how outlying reaction times on the task are identified and handled (e.g. individual reaction times > 3 standard deviations from the mean are removed from the distribution; removal of all participant data if > 25% errors are made). We used specification curve analysis across two alcohol Stroop datasets using alcohol-related stimuli (one published and one novel) to examine the robustness of the alcohol Stroop effect to different analytical decisions. We used a prior review of this research area to identify 27 unique analysis pipelines. Across both data sets, the pattern of results was similar. The alcohol Stroop effect was present and largely robust to different analysis pipelines. Increased variability in the Stroop effect was observed when implementing outlier cut-offs for individual reaction times, rather than the removal of participants. Stricter outlier thresholds tended to reduce the size of the Stroop interference effect. These specification curve analyses are the first to examine the robustness of the alcohol Stroop to different analysis strategies, and we encourage researchers to adopt such analytical methods to increase confidence in their inferences across cognitive and addiction science.

Exploring the intra-individual reliability of tDCS: A registered report

Transcranial direct current stimulation (tDCS), a form of non-invasive brain stimulation, has become an important tool for the study of in-vivo brain function due to its modulatory effects. Over the past two decades, interest in the influence of tDCS on behaviour has increased markedly, resulting in a large body of literature spanning multiple domains. However, the effect of tDCS on human performance often varies, bringing into question the reliability of this approach. While reviews and meta-analyses highlight the contributions of methodological inconsistencies and individual differences, no published studies have directly tested the intra-individual reliability of tDCS effects on behaviour. Here, we conducted a large scale, double-blinded, sham-controlled registered report to assess the reliability of two single-session low-dose tDCS montages, previously found to impact response selection and motor learning operations, across two separate time periods. Our planned analysis found no evidence for either protocol being effective nor reliable. Post-hoc explorative analyses found evidence that tDCS influenced motor learning, but not response selection learning. In addition, the reliability of motor learning performance across trials was shown to be disrupted by tDCS. These findings are amongst the first to shed light specifically on the intra-individual reliability of tDCS effects on behaviour and provide valuable information to the field.

An evaluation of the replicability of analyses using synthetic health data

Synthetic data generation is being increasingly used as a privacy preserving approach for sharing health data. In addition to protecting privacy, it is important to ensure that generated data has high utility. A common way to assess utility is the ability of synthetic data to replicate results from the real data. Replicability has been defined using two criteria: (a) replicate the results of the analyses on real data, and (b) ensure valid population inferences from the synthetic data. A simulation study using three heterogeneous real-world datasets evaluated the replicability of logistic regression workloads. Eight replicability metrics were evaluated: decision agreement, estimate agreement, standardized difference, confidence interval overlap, bias, confidence interval coverage, statistical power, and precision (empirical SE). The analysis of synthetic data used a multiple imputation approach whereby up to 20 datasets were generated and the fitted logistic regression models were combined using combining rules for fully synthetic datasets. The effects of synthetic data amplification were evaluated, and two types of generative models were used: sequential synthesis using boosted decision trees and a generative adversarial network (GAN). Privacy risk was evaluated using a membership disclosure metric. For sequential synthesis, adjusted model parameters after combining at least ten synthetic datasets gave high decision and estimate agreement, low standardized difference, as well as high confidence interval overlap, low bias, the confidence interval had nominal coverage, and power close to the nominal level. Amplification had only a marginal benefit. Confidence interval coverage from a single synthetic dataset without applying combining rules were erroneous, and statistical power, as expected, was artificially inflated when amplification was used. Sequential synthesis performed considerably better than the GAN across multiple datasets. Membership disclosure risk was low for all datasets and models. For replicable results, the statistical analysis of fully synthetic data should be based on at least ten generated datasets of the same size as the original whose analyses results are combined. Analysis results from synthetic data without applying combining rules can be misleading. Replicability results are dependent on the type of generative model used, with our study suggesting that sequential synthesis has good replicability characteristics for common health research workloads.

Exposing omitted moderators: Explaining why effect sizes differ in the social sciences

Policymakers increasingly rely on behavioral science in response to global challenges, such as climate change or global health crises. But applications of behavioral science face an important problem: Interventions often exert substantially different effects across contexts and individuals. We examine this heterogeneity for different paradigms that underlie many behavioral interventions. We study the paradigms in a series of five preregistered studies across one in-person and 10 online panels, with over 11,000 respondents in total. We find substantial heterogeneity across settings and paradigms, apply techniques for modeling the heterogeneity, and introduce a framework that measures typically omitted moderators. The framework's factors (Fluid Intelligence, Attentiveness, Crystallized Intelligence, and Experience) affect the effectiveness of many text-based interventions, producing different observed effect sizes and explaining variations across samples. Moderators are associated with effect sizes through two paths, with the intensity of the manipulation and with the effect of the manipulation directly. Our results motivate observing these moderators and provide a theoretical and empirical framework for understanding and predicting varying effect sizes in the social sciences.

The transparency of quantitative empirical legal research published in highly ranked law journals (2018-2020): an observational study

Background

Scientists are increasingly concerned with making their work easy to verify and build upon. Associated practices include sharing data, materials, and analytic scripts, and preregistering protocols. This shift towards increased transparency and rigor has been referred to as a "credibility revolution." The credibility of empirical legal research has been questioned in the past due to its distinctive peer review system and because the legal background of its researchers means that many often are not trained in study design or statistics. Still, there has been no systematic study of transparency and credibility-related characteristics of published empirical legal research.

Methods

To fill this gap and provide an estimate of current practices that can be tracked as the field evolves, we assessed 300 empirical articles from highly ranked law journals including both faculty-edited journals and student-edited journals.

Results

We found high levels of article accessibility (86%, 95% CI = [82%, 90%]), especially among student-edited journals (100%). Few articles stated that a study's data are available (19%, 95% CI = [15%, 23%]). Statements of preregistration (3%, 95% CI = [1%, 5%]) and availability of analytic scripts (6%, 95% CI = [4%, 9%]) were very uncommon. (i.e., they collected new data using the study's reported methods, but found results inconsistent or not as strong as the original).

Conclusion

We suggest that empirical legal researchers and the journals that publish their work cultivate norms and practices to encourage research credibility. Our estimates may be revisited to track the field's progress in the coming years.

Comparing data quality from an online and in-person lab sample on dynamic theory of mind tasks

Nearly half the published research in psychology is conducted with online samples, but the preponderance of these studies rely primarily on self-report measures. The current study validated data quality from an online sample on a novel, dynamic task by comparing performance between an in-lab and online sample on two dynamic measures of theory of mind-the ability to infer others' mental states. Theory of mind is a cognitively complex construct that has been widely studied across multiple domains of psychology. One task was based on the show The Office®, and has been previously validated by the authors with in-lab samples. The second was a novel task based on the show Nathan for You®, which was selected to account for familiarity effects associated with The Office. Both tasks measured various dimensions of theory of mind (inferring beliefs, understanding motivations, detecting deception, identifying faux pas, and understanding emotions). The in-person lab samples (N = 144 and 177, respectively) completed the tasks between-subject, whereas the online sample (N = 347 from Prolific Academic) completed them within-subject, with order counterbalanced. The online sample's performance across both tasks was reliable (Cronbach's α = .66). For The Office, the in-person sample outperformed the online sample on some types of theory of mind, but this was driven by their greater familiarity with the show. Indeed, for the relatively unfamiliar show Nathan for You, performance did not differ between the two samples. Together, these results suggest that crowdsourcing platforms elicit reliable performance on novel, dynamic, complex tasks.

Publication bias in psychology: A closer look at the correlation between sample size and effect size

Previously observed negative correlations between sample size and effect size (n-ES correlation) in psychological research have been interpreted as evidence for publication bias and related undesirable biases. Here, we present two studies aimed at better understanding to what extent negative n-ES correlations reflect such biases or might be explained by unproblematic adjustments of sample size to expected effect sizes. In Study 1, we analysed n-ES correlations in 150 meta-analyses from cognitive, organizational, and social psychology and in 57 multiple replications, which are free from relevant biases. In Study 2, we used a random sample of 160 psychology papers to compare the n-ES correlation for effects that are central to these papers and effects selected at random from these papers. n-ES correlations proved inconspicuous in meta-analyses. In line with previous research, they do not suggest that publication bias and related biases have a strong impact on meta-analyses in psychology. A much higher n-ES correlation emerged for publications' focal effects. To what extent this should be attributed to publication bias and related biases remains unclear.

Imagine Being Humble: Integrating Imagined Intergroup Contact and Cultural Humility to Foster Inclusive Intergroup Relations

To reduce prejudice and to promote intergroup harmony and equality, the imagined intergroup contact technique, based on the mental simulation of an encounter with an outgroup member, has been proposed. Though a substantial body of research has provided support for the efficacy of imagined intergroup contact in prejudice reduction, an alternative strand of research has raised questions about its effectiveness. In this experiment, we combined imagined intergroup contact with cultural humility, that is, an other-oriented, humble approach toward people with different cultural backgrounds, recognizing status and power imbalances and privileges. Specifically, we tested whether instructions aimed at eliciting cultural humility during imagined contact boosted its effectiveness in reducing prejudice and promoting future contact intentions, compared to a standard imagined contact condition and to a control imagination task. Intergroup anxiety was tested as a mediator of the effects of culturally humble imagined contact on reduced prejudice and on future contact intentions. We found that culturally humble imagined contact, compared to the two other conditions, reduced intergroup anxiety and yielded indirect effects on reduced prejudice and increased future contact intentions. The findings will be discussed by focusing on the integration of cultural humility in prejudice reduction techniques based on intergroup contact.

Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

Power analysis currently dominates sample size determination for experiments, particularly in grant and ethics applications. Yet, this focus could paradoxically result in suboptimal study design because publication biases towards studies with the largest effects can lead to the overestimation of effect sizes. In this Essay, we propose a paradigm shift towards better study designs that focus less on statistical power. We also advocate for (pre)registration and obligatory reporting of all results (regardless of statistical significance), better facilitation of team science and multi-institutional collaboration that incorporates heterogenization, and the use of prospective and living meta-analyses to generate generalizable results. Such changes could make science more effective and, potentially, more equitable, helping to cultivate better collaborations.

Hotels re-explored: Experience and influence of reciprocity and social normative appeals

In this paper we report two high-powered and pre-registered experiments, testing the robustness and conceptual development of reciprocity and social norm appeals. Both experiments assessed both psychological processes for complying with these appeals and pro-environmental behavioral intention in tourism settings. In Experiment 1 (N = 2004), participants reported lower psychological reactance levels after learning that the hotel engaged in resource conservation (i.e., indirect homeomorphic reciprocity). No statistically significant effect was obtained for either obligatory motivation, prosocial motivation, skepticism, or behavioral intentions to reuse hotel towels. Importantly, high baseline intention of reusing hotel towels might have limited the effect of appeals. Therefore, we targeted meat consumption in Experiment 2 (n = 2540). Results first showed stronger obligatory and prosocial motivation for all three reciprocity appeals, compared to the standard appeal. No statistically significant results were found for either reactance or skepticism. Finally, after learning that the hotel had made a financial contribution to an environmental organization (i.e., indirect heteromorphic reciprocity) participants showed reduced meat consumption intentions compared to the standard appeal. Overall, the results provide initial evidence for conceptually refining the norm of reciprocity to encourage pro-environmental behaviors and for understanding the underlying psychological processes.

The pupil collaboration: A multi-lab, multi-method analysis of goal attribution in infants

The rise of pupillometry in infant research over the last decade is associated with a variety of methods for data preprocessing and analysis. Although pupil diameter is increasingly recognized as an alternative measure of the popular cumulative looking time approach used in many studies (Jackson & Sirois, 2022), an open question is whether the many approaches used to analyse this variable converge. To this end, we proposed a crowdsourced approach to pupillometry analysis. A dataset from 30 9-month-old infants (15 girls; Mage = 282.9 days, SD = 8.10) was provided to 7 distinct teams for analysis. The data were obtained from infants watching video sequences showing a hand, initially resting between two toys, grabbing one of them (after Woodward, 1998). After habituation, infants were shown (in random order) a sequence of four test events that varied target position and target toy. Results show that looking times reflect primarily the familiar path of the hand, regardless of target toy. Gaze data similarly show this familiarity effect of path. The pupil dilation analyses show that features of pupil baseline measures (duration and temporal location) as well as data retention variation (trial and/or participant) due to different inclusion criteria from the various analysis methods are linked to divergences in findings. Two of the seven teams found no significant findings, whereas the remaining five teams differ in the pattern of findings for main and interaction effects. The discussion proposes guidelines for best practice in the analysis of pupillometry data.

Eleven years of student replication projects provide evidence on the correlates of replicability in psychology

Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of 170   replications . We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.

In models we trust: preregistration, large samples, and replication may not suffice

Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity.

The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward

The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a "crisis of confidence" stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current "crisis of confidence" in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.

Research Into Evidence-Based Psychological Interventions Needs a Stronger Focus on Replicability

Background

It is a precondition for evidence-based practice that research is replicable in a wide variety of clinical settings. Current standards for identifying evidence-based psychological interventions and making recommendations for clinical practice in clinical guidelines include criteria that are relevant for replicability, but a better understanding as well refined definitions of replicability are needed enabling empirical research on this topic. Recent advances on this issue were made in the wider field of psychology and in other disciplines, which offers the opportunity to define and potentially increase replicability also in research on psychological interventions.

Method

This article proposes a research strategy for assessing, understanding, and improving replicability in research on psychological interventions.

Results/Conclusion

First, we establish a replication taxonomy ranging from direct to conceptual replication adapted to the field of research on clinical interventions, propose study characteristics that increase the trustworthiness of results, and define statistical criteria for successful replication with respect to the quantitative outcomes of the original and replication studies. Second, we propose how to establish such standards for future research, i.e., in order to design future replication studies for psychological interventions as well as to apply them when investigating which factors are causing the (non-)replicability of findings in the current literature.

The validation of online webcam-based eye-tracking: The replication of the cascade effect, the novelty preference, and the visual world paradigm

The many benefits of online research and the recent emergence of open-source eye-tracking libraries have sparked an interest in transferring time-consuming and expensive eye-tracking studies from the lab to the web. In the current study, we validate online webcam-based eye-tracking by conceptually replicating three robust eye-tracking studies (the cascade effect, n = 134, the novelty preference, n = 45, and the visual world paradigm, n = 32) online using the participant's webcam as eye-tracker with the WebGazer.js library. We successfully replicated all three effects, although the effect sizes of all three studies shrank by 20-27%. The visual world paradigm was conducted both online and in the lab, using the same participants and a standard laboratory eye-tracker. The results showed that replication per se could not fully account for the effect size shrinkage, but that the shrinkage was also due to the use of online webcam-based eye-tracking, which is noisier. In conclusion, we argue that eye-tracking studies with relatively large effects that do not require extremely high precision (e.g., studies with four or fewer large regions of interest) can be done online using the participant's webcam. We also make recommendations for how the quality of online webcam-based eye-tracking could be improved.

Reproducibility in Neuroimaging Analysis: Challenges and Solutions

Recent years have marked a renaissance in efforts to increase research reproducibility in psychology, neuroscience, and related fields. Reproducibility is the cornerstone of a solid foundation of fundamental research-one that will support new theories built on valid findings and technological innovation that works. The increased focus on reproducibility has made the barriers to it increasingly apparent, along with the development of new tools and practices to overcome these barriers. Here, we review challenges, solutions, and emerging best practices with a particular emphasis on neuroimaging studies. We distinguish 3 main types of reproducibility, discussing each in turn. Analytical reproducibility is the ability to reproduce findings using the same data and methods. Replicability is the ability to find an effect in new datasets, using the same or similar methods. Finally, robustness to analytical variability refers to the ability to identify a finding consistently across variation in methods. The incorporation of these tools and practices will result in more reproducible, replicable, and robust psychological and brain research and a stronger scientific foundation across fields of inquiry.

Utility-value change and the role of emotional cost in video-based learning: a matter of student teachers' interpretation of experience

Motivational and emotional characteristics influence teachers' reflections on video clips from their own teaching. However, utility values and the role of emotional cost have not been considered yet in video-based research in teacher education. In the present study, 102 student teachers were randomly assigned to an intervention group (IG) with video-based documentation of a lesson and systematic writing assignment or to a control group (CG) with protocol-based documentation of a lesson and writing assignment. Multigroup latent change score analysis indicated that IG participants, on average, showed a 0.52 SDs higher increase in utility values than the CG three months after the teaching practicum. Emotional cost was negatively related to baseline utility-value scores but predicted latent change scores positively after the writing assignment. The study adds to the current repertoire of video-based learning opportunities by providing a systematic writing assignment targeting student teachers' interpretation of experiences to leverage motivation.

A Review of Multisite Replication Projects in Social Psychology: Is It Viable to Sustain Any Confidence in Social Psychology's Knowledge Base?

Multisite (multilab/many-lab) replications have emerged as a popular way of verifying prior research findings, but their record in social psychology has prompted distrust of the field and a sense of crisis. We review all 36 multisite social-psychology replications (plus three articles reporting multiple ministudies). We start by assuming that both the original and the multisite replications were conducted in honest and diligent fashion, despite often yielding different conclusions. Four of the 36 (11%) were clearly successful in terms of providing significant support for the original hypothesis, and five others (14%) had mixed results. The remaining 27 (75%) were failures. Multiple explanations for the generally poor record of replications are considered, including the possibility that the original hypothesis was wrong; operational failure; low engagement of participants; and bias toward failure. The relevant evidence is assessed as well. There was evidence for each of the possibilities listed above, with low engagement emerging as a widespread problem (reflected in high rates of discarded data and weak manipulation checks). The few procedures with actual interpersonal interaction fared much better than others. We discuss implications in relation to manipulation checks, effect sizes, and impact on the field and offer recommendations for improving future multisite projects.

Perception and action as viewed from the Theory of Event Coding: a multi-lab replication and effect size estimation of common experimental designs

The Theory of Event Coding (TEC) has influenced research on action and perception across the past two decades. It integrates several seminal empirical phenomena and it has continued to stimulate novel experimental approaches on the representational foundations of action control and perceptual experience. Yet, many of the most notable results surrounding TEC originate from an era of psychological research that relied on rather small sample sizes as judged by today's standards. This state hampers future research aiming to build on previous phenomena. We, therefore, provide a multi-lab re-assessment of the following six classical observations: response-effect compatibility, action-induced blindness, response-effect learning, stimulus-response binding, code occupation, and short-term response-effect binding. Our major goal is to provide precise estimates of corresponding effect sizes to facilitate future scientific endeavors. These effect sizes turned out to be considerably smaller than in the original reports, thus allowing for informed decisions on how to address each phenomenon in future work. Of note, the most relevant results of the original observations were consistently obtained in the present experiments as well.

Predicting and reasoning about replicability using structured groups

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

Vice Explanations for Conspiracism, Fundamentalism, and Extremism

In the literature on conspiracism, fundamentalism, and extremism, we find so-called vice explanations for the extreme behavior and extreme beliefs that they involve. These are explanations in terms of people's character traits, like arrogance, vengefulness, closed-mindedness, and dogmatism. However, such vice explanations face the so-called situationist challenge, which argues based on various experiments that either there are no vices or that they are not robust. Behavior and belief, so is the idea, are much better explained by appeal to numerous situational factors, like one's mood or how orderly one's environment is. This paper explores the situationist challenge to vice explanations for conspiracism, fundamentalism, and extremism in more detail by assessing the empirical evidence, analyzing the argumentation based on it, and drawing conclusions for where this leaves vice explanations. The main conclusion is that vice explanations for such extreme behavior and extreme beliefs need to be fine-tuned on various points, but that there is no reason to think that they have been discredited by empirical evidence. Moreover, the situationist challenge shows that sensitivity is needed for distinguishing when vice explanations for conspiracism, fundamentalism, and extremism are appropriate, when appeal to situational factors is more fitting, and when the two can be combined.

The competitive advantage of sanctioning institutions revisited: A multilab replication

Is peer sanctioning a sustainable solution to the problem of human cooperation? We conducted an exact multilab replication (N = 1,008; 7 labs × 12 groups × 12 participants) of an experiment by Gürerk, Irlenbusch, and Rockenbach published in Science in 2006 (Gürerk Ö, Irlenbusch B, Rockenbach B. The competitive advantage of sanctioning institutions. 2006. Science. 312(5770):108-111). In GIR2006 (N = 84; 1 lab × 7 groups × 12 participants), groups that allowed members to reward cooperators and punish defectors were found to outgrow and outperform groups without a peer-sanctioning institution. We find GIR2006 replicated in accordance with all preregistered replication criteria in five of the seven labs we sampled. There, the majority of participants joined groups with a sanctioning institution, and participants cooperated and profited more on average than in groups without a sanctioning institution. In the two other labs, results were weaker but still favored sanctioning institutions. These findings establish the competitive advantage of sanctioning institutions as a robust phenomenon within the European context.

On the Accuracy of Replication Failure Rates

A prominent approach to studying the replication crisis has been to conduct replications of several different scientific findings as part of the same research effort. The reported proportion of findings that these programs determined failed to replicate have become important statistics in the replication crisis. However, these "failure rates" are based on decisions about whether individual studies replicated, which are themselves subject to statistical uncertainty. In this article, we examine how that uncertainty impacts the accuracy of reported failure rates and find that the reported failure rates can be substantially biased and highly variable. Indeed, very high or very low failure rates could arise from chance alone.

Reproducibility of infant fNIRS studies: a meta-analytic approach

SIGNIFICANCE

Concerns about the reproducibility of experimental findings have recently emerged in many disciplines, from psychology to medicine and neuroscience. As NIRS is a relatively recent brain imaging technique, the question of reproducibility has not yet been systematically addressed.

AIM

The current study seeks to test the replicability of effects observed in NIRS experiments assessing young infants' rule-learning ability.

APPROACH

We conducted meta-analyses and mixed-effects modeling-based inferential statistics to determine whether effect sizes were replicable and comparable in a sample of 23 NIRS studies investigating infants' abilities to process repetition- and diversity-based regularities in linguistic and nonlinguistic auditory and visual sequences. Additionally, we tested whether effect sizes were modulated by different factors such as the age of participants or the laboratory. We obtained NIRS data from 12 published and 11 unpublished studies. The 23 studies involved a total of 487 infants, aged between 0 and 9 months, tested in four different countries (Canada, France, Italy, and USA).

RESULTS

Our most important finding is that study and laboratory were never significant moderators of variation in effect sizes, indicating that results replicated reliably across the different studies and labs included in the sample. We observed small-to-moderate effect sizes, similar to effect sizes found with other neuroimaging and behavioral techniques in the developmental literature. In line with existing findings, effect sizes were modulated by the participants' age and differed across the different regularities tested, with repetition-based regularities giving rise to the strongest effects; in particular, the overall magnitude of this effect in the left temporal region was 0.27 when analyzing the entire dataset.

CONCLUSIONS

Meta-analysis is a useful tool for assessing replicability and cross-study variability. Here, we have shown that infant NIRS studies in the language domain replicate robustly across various NIRS machines, testing sites, and developmental populations.

Using functional near-infrared spectroscopy to study the early developing brain: future directions and new challenges

Significance

Functional near-infrared spectroscopy (fNIRS) is a frequently used neuroimaging tool to explore the developing brain, particularly in infancy, with studies spanning from birth to toddlerhood (0 to 2 years). We provide an overview of the challenges and opportunities that the developmental fNIRS field faces, after almost 25 years of research.

Aim

We discuss the most recent advances in fNIRS brain imaging with infants and outlines the trends and perspectives that will likely influence progress in the field in the near future.

Approach

We discuss recent progress and future challenges in various areas and applications of developmental fNIRS from methodological and technological innovations to data processing and statistical approaches.

Results and Conclusions

The major trends identified include uses of fNIRS "in the wild," such as global health contexts, home and community testing, and hyperscanning; advances in hardware, such as wearable technology; assessment of individual variation and developmental trajectories particularly while embedded in studies examining other environmental, health, and context specific factors and longitudinal designs; statistical advances including resting-state network and connectivity, machine learning and reproducibility, and collaborative studies. Standardization and larger studies have been, and will likely continue to be, a major goal in the field, and new data analysis techniques, statistical methods, and collaborative cross-site projects are emerging.

Comparing Foodie Calls in Poland, the United Kingdom, and the United States: A Registered Replication Report

Collisson et al. (2020) found Dark Triad traits and gender role beliefs predicted "foodie calls," a phenomenon where people go on a date with others, to whom they are not attracted, for a free meal. Because gender roles and dating norms differ across cultures, we conducted a registered replication across different cultures by surveying 1838 heterosexual women from Poland, the United Kingdom (UK), and the United States (US). Relying on the structural equation modeling, as conducted in the original study, our findings revealed gender role beliefs best predicted foodie calls and their perceived acceptability, whereas the Dark Triad's general factor was nonsignificant. Analyses at the country level yielded mixed results. The original findings were replicated in the UK and Poland, but not in the US, where only narcissism predicted foodie calls. In the US, gender role beliefs predicted foodie call acceptability, but the Dark Triad general factor did not. Potential reasons for why traditional gender roles, but not the Dark Triad, predicted foodie calls in the US are discussed.

Association between personality traits and placebo effects: a preregistered systematic review and meta-analysis

ABSTRACT

Placebo effects are ubiquitous yet highly variable between individuals and therefore strongly affect clinical trial outcomes such as pain relief. It is unclear whether dispositional psychological traits influence responsiveness to placebo. This preregistered meta-analysis and systematic review synthesized the literature investigating the association between personality traits and placebo effects. Based on 21 studies with 798 participants, we performed formal meta-analyses for 10 different personality traits, including behavioral inhibition, fun seeking, goal-drive persistence, reward responsiveness, empathic concern, empathic fantasy, perspective-taking, personal distress, optimism, and anxiety. We did not find evidence of associations between any of these traits and magnitude of placebo effects, which was supported by equivalence tests. Furthermore, we did not find evidence for moderating factors such as placebo manipulation type (conditioning or nonconditioning) or condition (pain or nonpain). These findings challenge the notion that personality influences responsiveness to placebos and contradict its utility for identifying placebo "responders" and "nonresponders."

The effect of pre-event instructions on eyewitness identification

Research on eyewitness identification often involves exposing participants to a simulated crime and later testing memory using a lineup. We conducted a systematic review showing that pre-event instructions, instructions given before event exposure, are rarely reported and those that are reported vary in the extent to which they warn participants about the nature of the event or tasks. At odds with the experience of actual witnesses, some studies use pre-event instructions explicitly warning participants of the upcoming crime and lineup task. Both the basic and applied literature provide reason to believe that pre-event instructions may affect eyewitness identification performance. In the current experiment, we tested the impact of pre-event instructions on lineup identification decisions and confidence. Participants received non-specific pre-event instructions (i.e., "watch this video") or eyewitness pre-event instructions (i.e., "watch this crime video, you'll complete a lineup later") and completed a culprit-absent or -present lineup. We found no support for the hypothesis that participants who receive eyewitness pre-event instructions have higher discriminability than participants who receive non-specific pre-event instructions. Additionally, confidence-accuracy calibration was not significantly different between conditions. However, participants in the eyewitness condition were more likely to see the event as a crime and to make an identification than participants in the non-specific condition. Implications for conducting and interpreting eyewitness identification research and the basic research on instructions and attention are discussed.

Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project

The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407-425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project.

Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors

The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.

Replication in Evidence-Based Aphasia Treatments

PURPOSE

A critical review was completed to evaluate replication of aphasia treatments that have been vetted and accepted on the American Speech-Language-Hearing Association Practice Portal.

METHOD

The electronic databases Academic Search Premier, ProQuest Central, CINAHL Complete, and ERIC were searched for relevant articles using treatment names as keywords. Coders compared stimuli, material, design, and statistical analysis to pilot treatment approaches. Each study was coded as direct, conceptual, failed, or no replication.

RESULTS

Eighteen treatment approaches were selected for this review. A total of 188 articles met the inclusion criteria. Results revealed that 14 out of 18 treatment approaches were somehow replicated. Direct replications as the most valuable replication type for affirming previously found results were represented in only 1.5% of all studies (3/188). Failed direct replication were at 2% overall (4/188). Conceptual replications were more common across treatment approaches, but also represented at a low level with 22.8% (43/188). The majority of studies were coded as no replication attempt with 73.4%.

DISCUSSION

A critical factor in developing an evidence-based practice is the existence of replicated results for treatment. Replication evaluates the reproducibility of an author's or authors' research methodology and resulting outcomes and helps to ensure that observed treatment effects are reproducible. For an evidence-based treatment to be implemented or used in any clinical setting, it must be one that can be replicated. Direct and conceptual replications of aphasia treatment approaches were found to be alarmingly low considering the importance of replication in our field. It is recommended that replication should become more valued and mainstreamed in aphasia research. A replication database that compiles and maintains treatment manuals for replication purposes can increase the accessibility and acceptability of replications for researchers.

Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

BACKGROUND

Preregistration, the open science practice of specifying and registering details of a planned study prior to knowing the data, increases the transparency and reproducibility of research. Large-scale replication attempts for psychological results yielded shockingly low success rates and contributed to an increasing demand for open science practices among psychologists. However, preregistering one's studies is still not the norm in the field. Here, we conducted a study to explore possible reasons for this discrepancy.

METHODS

In a mixed-methods approach, we conducted an online survey assessing attitudes, motivations, and perceived obstacles with respect to preregistration. Respondents (N = 289) were psychological researchers that were recruited through their publications on Web of Science, PubMed, PSYNDEX, and PsycInfo, and preregistrations on OSF Registries. Based on the theory of planned behavior, we predicted that positive attitudes (moderated by the perceived importance of preregistration) as well as a favorable subjective norm and higher perceived behavioral control positively influence researchers' intention to preregister (directional hypothesis 1). Furthermore, we expected an influence of research experience on attitudes and perceived motivations and obstacles regarding preregistration (non-directional hypothesis 2). We analyzed these hypotheses with multiple regression models and included preregistration experience as a control variable.

RESULTS

Researchers' attitudes, subjective norms, perceived behavioral control, and the perceived importance of preregistration significantly predicted researchers' intention to use preregistration in the future (see hypothesis 1). Research experience influenced both researchers' attitudes and their perception of motivations to preregister, but not the perception of obstacles (see hypothesis 2). Descriptive reports on researchers' attitudes, motivations and obstacles regarding preregistration are provided.

DISCUSSION

Many researchers had already preregistered and had a rather positive attitude toward preregistration. Nevertheless, several obstacles were identified that may be addressed to improve and foster preregistration.

A practical guide for studying human behavior in the lab

In the last few decades, the field of neuroscience has witnessed major technological advances that have allowed researchers to measure and control neural activity with great detail. Yet, behavioral experiments in humans remain an essential approach to investigate the mysteries of the mind. Their relatively modest technological and economic requisites make behavioral research an attractive and accessible experimental avenue for neuroscientists with very diverse backgrounds. However, like any experimental enterprise, it has its own inherent challenges that may pose practical hurdles, especially to less experienced behavioral researchers. Here, we aim at providing a practical guide for a steady walk through the workflow of a typical behavioral experiment with human subjects. This primer concerns the design of an experimental protocol, research ethics, and subject care, as well as best practices for data collection, analysis, and sharing. The goal is to provide clear instructions for both beginners and experienced researchers from diverse backgrounds in planning behavioral experiments.

Proposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science

INTRODUCTION

To improve the rigor of science, experimental evidence for scientific claims ideally needs to be replicated repeatedly with comparable analyses and new data to increase the collective confidence in the veracity of those claims. Large replication projects in psychology and cancer biology have evaluated the replicability of their fields but no collaborative effort has been undertaken in sports and exercise science. We propose to undertake such an effort here. As this is the first large replication project in this field, there is no agreed-upon protocol for selecting studies to replicate. Criticism of previous selection protocols include claims they were non-randomised and non-representative. Any selection protocol in sports and exercise science must be representative to provide an accurate estimate of replicability of the field. Our aim is to produce a protocol for selecting studies to replicate for inclusion in a large replication project in sports and exercise science.

METHODS

The proposed selection protocol uses multiple inclusion and exclusion criteria for replication study selection, including: the year of publication and citation rankings, research disciplines, study types, the research question and key dependent variable, study methods and feasibility. Studies selected for replication will be stratified into pools based on instrumentation and expertise required, and will then be allocated to volunteer laboratories for replication. Replication outcomes will be assessed using a multiple inferential strategy and descriptive information will be reported regarding the final number of included and excluded studies, and original author responses to requests for raw data.

Open science, the replication crisis, and environmental public health

Concerns about a crisis of mass irreplicability across scientific fields ("the replication crisis") have stimulated a movement for open science, encouraging or even requiring researchers to publish their raw data and analysis code. Recently, a rule at the US Environmental Protection Agency (US EPA) would have imposed a strong open data requirement. The rule prompted significant public discussion about whether open science practices are appropriate for fields of environmental public health. The aims of this paper are to assess (1) whether the replication crisis extends to fields of environmental public health; and (2) in general whether open science requirements can address the replication crisis. There is little empirical evidence for or against mass irreplicability in environmental public health specifically. Without such evidence, strong claims about whether the replication crisis extends to environmental public health - or not - seem premature. By distinguishing three concepts - reproducibility, replicability, and robustness - it is clear that open data initiatives can promote reproducibility and robustness but do little to promote replicability. I conclude by reviewing some of the other benefits of open science, and offer some suggestions for funding streams to mitigate the costs of adoption of open science practices in environmental public health.

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Impacts of Technology Use on the Workload of Registered Nurses: A Scoping Review

INTRODUCTION

Technology is an integral part of healthcare. With the rapid development of technological innovations that inform and support nurses, it is important to assess how these technologies may affect their workload particularly in rural contexts, where the workforce and supports may be limited.

METHODS

This literature review guided by Arksey and O'Malley's scoping review framework describes the breadth of technologies which impact on nurses' workload. Five databases (PubMed, CINAHL, PsycInfo, Web of Science, Business Source Complete) were searched. Thirty-five articles met the inclusion criteria. A data matrix was used to organize the findings.

FINDINGS

The technology interventions described in the articles covered diverse topics including: Cognitive care technologies; Healthcare providers' technologies; Communication technologies; E-learning technologies; and Assistive technologies and were categorized as: Digital Information Solutions; Digital Education; Mobile Applications; Virtual Communication; Assistive Devices; and Disease diagnoses groups based on the common features.

CONCLUSION

Technology can play an important role to support nurses working in rural areas, however, not all technologies have the same impact. While some technologies showed evidence to positively impact nursing workload, this was not universal. Technology solutions should be considered on a contextual basis and thought should be given when selecting technologies to support nursing workload.

Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field

Known methodological issues such as publication bias, questionable research practices and studies with underpowered designs are known to decrease the replicability of study findings. The presence of such issues has been widely established across different research fields, especially in psychology. Their presence raised the first concerns that the replicability of study findings could be low and led researchers to conduct large replication projects. These replication projects revealed that a significant portion of original study findings could not be replicated, giving rise to the conceptualization of the replication crisis. Although previous research in the field of sports and exercise science has identified the first warning signs, such as an overwhelming proportion of significant findings, small sample sizes and lack of data availability, their possible consequences for the replicability of our field have been overlooked. We discuss the consequences of the above issues on the replicability of our field and offer potential solutions to improve replicability.

The effects of base rate neglect on sequential belief updating and real-world beliefs

Base-rate neglect is a pervasive bias in judgment that is conceptualized as underweighting of prior information and can have serious consequences in real-world scenarios. This bias is thought to reflect variability in inferential processes but empirical support for a cohesive theory of base-rate neglect with sufficient explanatory power to account for longer-term and real-world beliefs is lacking. A Bayesian formalization of base-rate neglect in the context of sequential belief updating predicts that belief trajectories should exhibit dynamic patterns of dependence on the order in which evidence is presented and its consistency with prior beliefs. To test this, we developed a novel 'urn-and-beads' task that systematically manipulated the order of colored bead sequences and elicited beliefs via an incentive-compatible procedure. Our results in two independent online studies confirmed the predictions of the sequential base-rate neglect model: people exhibited beliefs that are more influenced by recent evidence and by evidence inconsistent with prior beliefs. We further found support for a noisy-sampling inference model whereby base-rate neglect results from rational discounting of noisy internal representations of prior beliefs. Finally, we found that model-derived indices of base-rate neglect-including noisier prior representation-correlated with propensity for unusual beliefs outside the laboratory. Our work supports the relevance of Bayesian accounts of sequential base-rate neglect to real-world beliefs and hints at strategies to minimize deleterious consequences of this pervasive bias.

Remarkably reproducible psychological (memory) phenomena in the classroom: some evidence for generality from small-N research

BACKGROUND

Mainstream psychology is experiencing a crisis of confidence. Many of the methodological solutions offered in response have focused largely on statistical alternatives to null hypothesis statistical testing, ignoring nonstatistical remedies that are readily available within psychology; namely, use of small-N designs. In fact, many classic memory studies that have passed the test of replicability used them. That methodological legacy warranted a retrospective look at nonexperimental data to explore the generality of the reported effects.

METHOD

Various classroom demonstrations were conducted over multiple semesters in introductory psychology courses with typical, mostly freshman students from a predominantly white private Catholic university in the US Midwest based on classic memory experiments on immediate memory span, chunking, and depth of processing.

RESULTS

Students tended to remember 7 ± 2 digits, remembered more digits of π following an attached meaningful story, and remembered more words after elaborative rehearsal than after maintenance rehearsal. These results amount to replications under uncontrolled classroom environments of the classic experiments originally conducted largely outside of null hypothesis statistical testing frameworks.

CONCLUSIONS

In light of the ongoing replication crisis in psychology, the results are remarkable and noteworthy, validating these historically important psychological findings. They are testament to the reliability of reproducible effects as the hallmark of empirical findings in science and suggest an alternative approach to commonly proffered solutions to the replication crisis.

Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines

The open and transparent documentation of scientific processes has been established as a core antecedent of free knowledge. This also holds for generating robust insights in the scope of research projects. To convince academic peers and the public, the research process must be understandable and retraceable (reproducible), and repeatable (replicable) by others, precluding the inclusion of fluke findings into the canon of insights. In this contribution, we outline what reproducibility and replicability (R&R) could mean in the scope of different disciplines and traditions of research and which significance R&R has for generating insights in these fields. We draw on projects conducted in the scope of the Wikimedia "Open Science Fellows Program" (Fellowship Freies Wissen), an interdisciplinary, long-running funding scheme for projects contributing to open research practices. We identify twelve implemented projects from different disciplines which primarily focused on R&R, and multiple additional projects also touching on R&R. From these projects, we identify patterns and synthesize them into a roadmap of how research projects can achieve R&R across different disciplines. We further outline the ground covered by these projects and propose ways forward.

The Dilemma of Reproducibility of Gating Isotherms for Flexible MOFs

Porous materials receive a high level of scientific and technological interest due to their applications in various fields such as adsorption, separation and storage, catalysis, ion exchange, nanotechnology, etc. Gas adsorption is a well-established tool for the characterization of the texture of porous solids. Physisorption isotherms are generally expected to be well reproducible for rigid adsorbents, but this is not always the case for nonrigid (flexible) materials. The presence of a metastability region and sensitivity of the activation barriers to the material's texture often influence the isotherms' run. Here, we address the complexity that arises in terms of reproducibility and sample handling for flexible metal-organic frameworks, with the example of DUT-8(Ni). It belongs to the group of "gate opening" metal-organic frameworks and is a typical representative of the pillared layer compounds. We propose characteristic parameters for the analysis and comparison of adsorption isotherms, showing the "gate opening" step, associated with the adsorption-induced solid-state phase transition. A set of 50 nitrogen physisorption isotherms measured at 77 K were analyzed and correlated with the synthetic and outgassing conditions. The study highlights the importance of accurate descriptions and record-keeping of experimental details and their role in the replication of scientific results.