The Psychological Science Accelerator's COVID-19 rapid-response dataset

In response to the COVID-19 pandemic, the Psychological Science Accelerator coordinated three large-scale psychological studies to examine the effects of loss-gain framing, cognitive reappraisals, and autonomy framing manipulations on behavioral intentions and affective measures. The data collected (April to October 2020) included specific measures for each experimental study, a general questionnaire examining health prevention behaviors and COVID-19 experience, geographical and cultural context characterization, and demographic information for each participant. Each participant started the study with the same general questions and then was randomized to complete either one longer experiment or two shorter experiments. Data were provided by 73,223 participants with varying completion rates. Participants completed the survey from 111 geopolitical regions in 44 unique languages/dialects. The anonymized dataset described here is provided in both raw and processed formats to facilitate re-use and further analyses. The dataset offers secondary analytic opportunities to explore coping, framing, and self-determination across a diverse, global sample obtained at the onset of the COVID-19 pandemic, which can be merged with other time-sampled or geographic data.

A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect

We conducted a preregistered multilaboratory project (k = 36; N = 3,531) to assess the size and robustness of ego-depletion effects using a novel replication method, termed the paradigmatic replication approach. Each laboratory implemented one of two procedures that was intended to manipulate self-control and tested performance on a subsequent measure of self-control. Confirmatory tests found a nonsignificant result (d = 0.06). Confirmatory Bayesian meta-analyses using an informed-prior hypothesis (δ = 0.30, SD = 0.15) found that the data were 4 times more likely under the null than the alternative hypothesis. Hence, preregistered analyses did not find evidence for a depletion effect. Exploratory analyses on the full sample (i.e., ignoring exclusion criteria) found a statistically significant effect (d = 0.08); Bayesian analyses showed that the data were about equally likely under the null and informed-prior hypotheses. Exploratory moderator tests suggested that the depletion effect was larger for participants who reported more fatigue but was not moderated by trait self-control, willpower beliefs, or action orientation.

A multi-country test of brief reappraisal interventions on emotions during the COVID-19 pandemic

The COVID-19 pandemic has increased negative emotions and decreased positive emotions globally. Left unchecked, these emotional changes might have a wide array of adverse impacts. To reduce negative emotions and increase positive emotions, we tested the effectiveness of reappraisal, an emotion-regulation strategy that modifies how one thinks about a situation. Participants from 87 countries and regions (n = 21,644) were randomly assigned to one of two brief reappraisal interventions (reconstrual or repurposing) or one of two control conditions (active or passive). Results revealed that both reappraisal interventions (vesus both control conditions) consistently reduced negative emotions and increased positive emotions across different measures. Reconstrual and repurposing interventions had similar effects. Importantly, planned exploratory analyses indicated that reappraisal interventions did not reduce intentions to practice preventive health behaviours. The findings demonstrate the viability of creating scalable, low-cost interventions for use around the world. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on 12 May 2020. The protocol, as accepted by the journal, can be found at https://doi.org/10.6084/m9.figshare.c.4878591.v1.

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

Many Labs 5: Registered Replication of Shnabel and Nadler (2008), Study 4

Shnabel and Nadler (2008) assessed a needs-based model of reconciliation suggesting that in conflicts, victims and perpetrators have different psychological needs that when satisfied increase the chances of reconciliation. For instance, Shnabel and Nadler found that after a conflict, perpetrators indicated that they had a need for social acceptance and were more likely to reconcile after their sense of social acceptance was restored, whereas victims indicated that they had a need for power and were more likely to reconcile after their sense of power was restored. Gilbert (2016), as a part of the Reproducibility Project: Psychology (RP:P), attempted to replicate these findings using different study materials but did not find support for the original effect. In an attempt to reconcile these discrepant findings, we conducted two new sets of replications—one using the RP:P protocol and another using modified materials meant to be more relatable to undergraduate participants. Teams from eight universities contributed to data collection ( N = 2,738). We did find moderation by protocol; the focal interaction from the revised protocol, but not from the RP:P protocol, replicated the interaction in the original study. We discuss differences in, and possible explanations for, the patterns of results across protocols.

Older and younger adults' interactions with friends and strangers in an iterated prisoner's dilemma

This study examined age-related differences in behavioral reactions to interpersonal conflict within an iterated prisoner's dilemma (PD). Participants completed an iterated PD game alone and with a partner, either a stranger or a friend who accompanied them to the session. The partner, however, was actually a program that occasionally behaved selfishly or always reciprocated. Afterwards, participants formed trait impressions of their partner's morality and competence. Participants cooperated more with friends than strangers and more with reciprocating partners than selfish ones. Older adults cooperated more with selfish partners and offered more favorable impressions than did younger adults. Overall, perceived partner trait morality was positively associated with cooperative behavior. Relative to younger adults, older adults were more passive during conflict but grew less so as selfishness continued. This passivity co-occurred with more favorable partner impressions and better objective performance, suggesting a degree of calibration not shown by younger adults.

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance ( p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion ( p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Investigating Variation in Replicability

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

Commentaries and Rejoinder on

While direct replications such as the “Many Labs” project are extremely valuable in testing the reliability of published findings across laboratories, they reflect the common reliance in psychology on single vignettes or stimuli, which limits the scope of the conclusions that can be reached. New experimental tools and statistical techniques make it easier to routinely sample stimuli, and to appropriately treat them as random factors. We encourage researchers to get into the habit of including multiple versions of the content (e.g., stimuli or vignettes) in their designs, to increase confidence in cross-stimulus generalization and to yield more realistic estimates of effect size. We call on editors to be aware of the challenges inherent in such stimulus sampling, to expect and tolerate unexplained variability in observed effect size between stimuli, and to encourage stimulus sampling instead of the deceptively cleaner picture offered by the current reliance on single stimuli.

Birth Order Has No Effect on Intelligence: A Reply and Extension of Previous Findings

We address points raised by Zajonc and Sulloway, who reject findings showing that birth order has no effect on intelligence. Many objections to findings of null birth-order results seem to stem from a misunderstanding of the difference between study designs where birth order is confounded with true causal influences on intelligence across families and designs that control for some of these influences. We discuss some of the consequences of not appreciating the nature of this difference. When between-family confounds are controlled using appropriate study designs and techniques such as multilevel modeling, birth order is shown not to influence intelligence. We conclude with an empirical investigation of the replicability and generalizability of this approach.

A multilevel approach to the relationship between birth order and intelligence

Many studies show relationships between birth order and intelligence but use cross-sectional designs or manifest other threats to internal validity. Multilevel analyses with a control variable show that when these threats are removed, two major results emerge: (a) birth order has no significant influence on children's intelligence and (b) earlier reported birth order effects on intelligence are attributable to factors that vary between, not within, families. Analyses on 7- to 8 - and 13- to 14-year-old children from the National Longitudinal Survey of Youth support these conclusions. When hierarchical data structures, age variance of children, and within-family versus between-family variance sources are taken into account, previous research is seen in a new light.

Perceived likelihood as a measure of optimism and pessimism: Support for the Future Events Scale

The Future Events Scale (FES; S. M. Andersen, 1990) is an expectancy-based measure of optimism and pessimism, grounded in cognitive theories of depression, with implications for clinical practice. Although ample research has documented the utility of the FES in predicting important cognitive and behavioral outcomes, psychometric data on the scale are lacking. The current article presents multisample analyses to show that the FES has clear factor structure, good reliability, and a theoretically meaningful nomological network. The FES is shown to be distinct from the best known measure of optimism and pessimism, the Life Orientation Test (M. F. Scheier & C. S. Carver, 1985). Applications are discussed.

The effect of lexical, pragmatic, and morphological violations on reading time and deviance ratings of English and German sentences

Effects on reading time and deviance ratings of word-choice violations were studied at the surface code, propositional textbase, and situation model levels of representation in German and English sentences. Lexical violations (propositional textbase level) such as The housewife massaged the bread dough were rated as more deviant from normal language than were pragmatic violations (situation model level) such as The police officer shot the parking violator, although the pragmatic violations took longer to comprehend. The addition of a morphological (surface code) violation through the wrong form of an article (e.g., a ugly car) decreased reading time but only in cases where that material was deeply processed and where the morphology carried substantial syntactic information (e.g., in German, the definite article carries case, gender, and number information). The results thus confirmed the operation of comprehension strategies at multiple levels of representation and identified some cross-linguistic generality in comprehension and some language differences.