Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

The impact of hunger on indulgent food choices is moderated by healthy eating concerns

Research has shown that hungry individuals are more impulsive, impatient, and prone to make indulgent food choices compared to their satiated counterparts. However, the literature is still mixed, with some studies showing such results while others fail to demonstrate hunger effects on consumers' choice behavior. The current cross-sectional study (N = 461) sought to address these inconsistencies by examining whether the link between hunger and people's propensity to make indulgent (vs. virtuous) food choices is moderated by their healthy eating concerns. Our findings revealed a weak but significant association between participants' self-reported hunger levels and their likelihood of making indulgent rather than virtuous food choices (e.g., preferring a chocolate cake instead of a fruit salad). Importantly, this effect was moderated by their healthy eating concerns, such that the link between hunger and choice likelihood of indulgent food options only emerged among participants who scored lower, but not higher, in healthy eating concerns. We also replicated these results in a robustness check that focused on the extent to which participants indicated having a healthy lifestyle (e.g., exercising regularly), with a similar moderating influence of this factor. Together, these findings shed light on the importance of considering certain boundary conditions for establishing a link between hunger and consumers' food choices, thus adding nuance to the growing body of hunger-related literature. The results emphasize the importance of ensuring the availability of healthier snack options in environments wherein foods and beverages can be consumed, particularly at times when consumers tend to be hungry, to promote healthier eating habits.

Factor Analysis With Unusual Sexual Interests: A Replication Study in a Representative Population Sample

A previous study found a variety of unusual sexual interests to cluster in a five-factor structure, namely submission/masochism, forbidden sexual activities, dominance/sadism, mysophilia, and fetishism (Schippers et al., 2021). The current study was an empirical replication to examine whether these findings generalized to a representative population sample. An online, anonymous sample (N = 256) representative of the Dutch adult male population rated 32 unusual sexual interests on a scale from 1 (very unappealing) to 7 (very appealing). An exploratory factor analysis assessed whether similar factors would emerge as in the original study. A subsequent confirmatory factor analysis served to confirm the factor structure. Four slightly different factors of sexual interest were found: extreme, illegal and mysophilic sexual activities; light BDSM without real pain or suffering; heavy BDSM that may include pain or suffering; and illegal but lower-sentenced and fetishistic sexual activities. The model fit was acceptable. The representative replication sample was more sexually conservative and showed less sexual engagement than the original convenience sample. On a fundamental level, sexual interest in light BDSM activities and extreme, forbidden, and mysophilic activities seem to be relatively separate constructs.

Sixty Years After Orne's American Psychologist Article: A Conceptual Framework for Subjective Experiences Elicited by Demand Characteristics

Study participants form beliefs based on cues present in a testing situation (demand characteristics). These beliefs can alter study outcomes (demand effects). Neglecting demand effects can threaten the internal and external validity of studies (including their replication). While demand characteristics garnered much attention following Orne's introduction of this notion, consideration of their effects has become sparse in experimental reports. Moreover, the concept remains confusing. Here, we introduce a conceptual framework for subjective experiences elicited by demand characteristics. The model distinguishes between participants' awareness of the hypothesis, their motivation to comply with it, and the strategy they use to meet situational requirements. We stress that demand characteristics can give rise to genuine experiences. To illustrate, we apply the model to Evaluative Conditioning and the Rubber Hand Illusion. In the General Discussion, we discuss risks and opportunities associated with demand characteristics, and we explain that they remain highly relevant to current research.

Why are people antiscience, and what can we do about it?

From vaccination refusal to climate change denial, antiscience views are threatening humanity. When different individuals are provided with the same piece of scientific evidence, why do some accept whereas others dismiss it? Building on various emerging data and models that have explored the psychology of being antiscience, we specify four core bases of key principles driving antiscience attitudes. These principles are grounded in decades of research on attitudes, persuasion, social influence, social identity, and information processing. They apply across diverse domains of antiscience phenomena. Specifically, antiscience attitudes are more likely to emerge when a scientific message comes from sources perceived as lacking credibility; when the recipients embrace the social membership or identity of groups with antiscience attitudes; when the scientific message itself contradicts what recipients consider true, favorable, valuable, or moral; or when there is a mismatch between the delivery of the scientific message and the epistemic style of the recipient. Politics triggers or amplifies many principles across all four bases, making it a particularly potent force in antiscience attitudes. Guided by the key principles, we describe evidence-based counteractive strategies for increasing public acceptance of science.

Accelerating Psychological Science With Metastudies: A Demonstration Using the Risky-Choice Framing Effect

A metastudy is a set of many tiny studies (microstudies) created from a much larger collection of possibilities. Metastudies can yield many of the benefits of time-consuming replications and meta-analyses but more efficiently and with greater attention to generalizability and the causal effects of moderators. Statistical precision and power are higher than in studies with the same total sample size but with fewer conditions and more participants per condition. In this article, we describe metastudies and their benefits, demonstrate how to conduct a metastudy using the well-known risky-choice framing effect as an example, and report on the generalizability of that effect. In three metastudies (total N = 2,338), the framing effect generalized well across most of the potential moderators tested, as was expected. Surprisingly, however, the effect was up to twice as large when the certain option was replaced with a slightly risky option; prospect theory predicts the opposite, and fuzzy-trace theory predicts no difference. Metastudies provide a relatively quick and not-so-painful way of examining an effect's generalizability without waiting for a meta-analysis. Both individual labs and multilab networks are encouraged to shift from traditional studies to metastudies.

Toward a cumulative science of vocal markers of autism: A cross-linguistic meta-analysis-based investigation of acoustic markers in American and Danish autistic children

Acoustic atypicalities in speech production are argued to be potential markers of clinical features in autism spectrum disorder (ASD). A recent meta-analysis highlighted shortcomings in the field, in particular small sample sizes and study heterogeneity. We showcase a cumulative (i.e., explicitly building on previous studies both conceptually and statistically) yet self-correcting (i.e., critically assessing the impact of cumulative statistical techniques) approach to prosody in ASD to overcome these issues. We relied on the recommendations contained in the meta-analysis to build and analyze a cross-linguistic corpus of multiple speech productions in 77 autistic and 72 neurotypical children and adolescents (>1000 recordings in Danish and US English). We used meta-analytically informed and skeptical priors, with informed priors leading to more generalizable inference. We replicated findings of a minimal cross-linguistically reliable distinctive acoustic profile for ASD (higher pitch and longer pauses) with moderate effect sizes. We identified novel reliable differences between the two groups for normalized amplitude quotient, maxima dispersion quotient, and creakiness. However, the differences were small, and there is likely no one acoustic profile characterizing all autistic individuals. We identified reliable relations of acoustic features with individual differences (age, gender), and clinical features (speech rate and ADOS sub-scores). Besides cumulatively building our understanding of acoustic atypicalities in ASD, the study shows how to use systematic reviews and meta-analyses to guide the design and analysis of follow-up studies. We indicate future directions: larger and more diverse cross-linguistic datasets, focus on heterogeneity, self-critical cumulative approaches, and open science. LAY SUMMARY: Autistic individuals are reported to speak in distinctive ways. Distinctive vocal production can affect social interactions and social development and could represent a noninvasive way to support the assessment of autism spectrum disorder (ASD). We systematically checked whether acoustic atypicalities highlighted in previous articles could be actually found across multiple recordings and two languages. We find a minimal acoustic profile of ASD: higher pitch, longer pauses, increased hoarseness and creakiness of the voice. However, there is much individual variability (by age, sex, language, and clinical characteristics). This suggests that the search for one common "autistic voice" might be naive and more fine-grained approaches are needed.

Mind-Body Practices and Self-Enhancement: Direct Replications of Gebauer et al.'s (2018) Experiments 1 and 2

Mind-body practices such as yoga and meditation are often believed to instill a "quiet ego," entailing less self-enhancement. In two experiments, however, Gebauer et al. (2018) demonstrated that mind-body practices may actually increase self-enhancement, particularly because such practices become self-central bases for self-esteem. We conducted preregistered replications of both of Gebauer et al.'s experiments. Experiment 1 was a field study of Canadian yoga students (N = 97), and Experiment 2 was a multiwave meditation intervention among Canadian university students (N = 300). Our results supported Gebauer et al.'s original conclusions that mind-body practices increase self-enhancement. Although the self-centrality effects were not clearly replicated in either experiment, we found evidence that measurement and sampling differences may explain this discrepancy. Moreover, an integrative data analysis of the original and the replication data strongly supported all of Gebauer et al.'s conclusions. In short, we provide new evidence against the ego-quieting perspective and in support of the self-centrality interpretation of mind-body practices.

Evaluating Research in Personality and Social Psychology: Considerations of Statistical Power and Concerns About False Findings

Traditionally, statistical power was viewed as relevant to research planning but not evaluation of completed research. However, following discussions of high false finding rates (FFRs) associated with low statistical power, the assumed level of statistical power has become a key criterion for research acceptability. Yet, the links between power and false findings are not as straightforward as described. Assumptions underlying FFR calculations do not reflect research realities in personality and social psychology. Even granting the assumptions, the FFR calculations identify important limitations to any general influences of statistical power. Limits for statistical power in inflating false findings can also be illustrated through the use of FFR calculations to (a) update beliefs about the null or alternative hypothesis and (b) assess the relative support for the null versus alternative hypothesis when evaluating a set of studies. Taken together, statistical power should be de-emphasized in comparison to current uses in research evaluation.

Context Dependency as a Predictor of Replicability

We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

Health information-seeking behavior in the time of COVID-19: information horizons methodology to decipher source path during a global pandemic

Purpose

To determine the differences, as represented by information horizons mapping, in the health information-seeking behavior from a group of participants between March 2019 and April 2020 of the novel coronavirus pandemic.

Design/methodology/approach

In March 2019, the author conducted a study on health information-seeking behavior in which 149 participants drew information horizons maps in a health-related context. They also took health and information literacy assessments. This exact study was replicated using the same population in April 2020 to determine the differences in what the participants drew on their maps and how these changes interacted with their health and information literacy, their age and their education.

Findings

There is a statistically significant difference in the increased number of sources and the ranked quality of the sources that people used during the pandemic. Participants were much more likely to use credible sources and news sources, especially if they were older, more educated and had higher literacy levels – both health and information. They also relied heavily on social media. The participant group in the pandemic had a much heavier reliance on sources that are often used in a passive encountering way but engaging with them in an active information-seeking manner. The health information-seeking behavior in this study did not adhere to other research that found issue with information overload, avoidance and cyberchondria in response to crisis situations.

Originality/value

This article utilizes information horizons methodology to explore pre- and post-pandemic information-seeking. It is completely unique in this approach.