How Far Does Stereotype Threat Reach? The Potential Detriment of Face Validity in Cognitive Ability Testing

Construct Validation of Experimental Manipulations in Social Psychology: Current Practices and Recommendations for the Future

Experimental manipulations in social psychology must exhibit construct validity by influencing their intended psychological constructs. Yet how do experimenters in social psychology attempt to establish the construct validity of their manipulations? Following a preregistered plan, we coded 348 experimental manipulations from the 2017 issues of the Journal of Personality and Social Psychology. Representing a reliance on "on-the-fly" experimentation, the vast majority of these manipulations were created ad hoc for a given study and were not previously validated before implementation. A minority of manipulations had their construct validity evaluated by pilot testing before implementation or via a manipulation check. Of the manipulation checks administered, most were face valid, single-item self-reports, and only a few met criteria for "true" validation. In aggregate, roughly two fifths of manipulations relied solely on face validity. To the extent that they are representative of the field, these results suggest that best practices for validating manipulations are not commonplace-a potential contributor to replicability issues. These issues can be remedied by validating manipulations before implementation using validated manipulation checks, standardizing manipulation protocols, estimating the size and duration of manipulations' effects, and estimating each manipulation's effects on multiple constructs within the target nomological network.

Can Stereotype Threat Explain the Gender Gap in Mathematics Performance and Achievement?

Men and women score similarly in most areas of mathematics, but a gap favoring men is consistently found at the high end of performance. One explanation for this gap, stereotype threat, was first proposed by Spencer, Steele, and Quinn (1999) and has received much attention. We discuss merits and shortcomings of this study and review replication attempts. Only 55% of the articles with experimental designs that could have replicated the original results did so. But half of these were confounded by statistical adjustment of preexisting mathematics exam scores. Of the unconfounded experiments, only 30% replicated the original. A meta-analysis of these effects confirmed that only the group of studies with adjusted mathematics scores displayed the stereotype threat effect. We conclude that although stereotype threat may affect some women, the existing state of knowledge does not support the current level of enthusiasm for this as a mechanism underlying the gender gap in mathematics. We argue there are many reasons to close this gap, and that too much weight on the stereotype explanation may hamper research and implementation of effective interventions.