Assume a can opener

We propose a friendly amendment to integrative experiment design (IED), adversarial-collaboration IED, that incentivizes research teams from competing theoretical perspectives to identify zones of the design space where they possess an explanatory edge. This amendment is especially critical in debates that have high policy stakes and carry a strong normative-political charge that might otherwise prevent free exchange of ideas.

Prosocial motives underlie scientific censorship by scientists: A perspective and research agenda

Science is among humanity's greatest achievements, yet scientific censorship is rarely studied empirically. We explore the social, psychological, and institutional causes and consequences of scientific censorship (defined as actions aimed at obstructing particular scientific ideas from reaching an audience for reasons other than low scientific quality). Popular narratives suggest that scientific censorship is driven by authoritarian officials with dark motives, such as dogmatism and intolerance. Our analysis suggests that scientific censorship is often driven by scientists, who are primarily motivated by self-protection, benevolence toward peer scholars, and prosocial concerns for the well-being of human social groups. This perspective helps explain both recent findings on scientific censorship and recent changes to scientific institutions, such as the use of harm-based criteria to evaluate research. We discuss unknowns surrounding the consequences of censorship and provide recommendations for improving transparency and accountability in scientific decision-making to enable the exploration of these unknowns. The benefits of censorship may sometimes outweigh costs. However, until costs and benefits are examined empirically, scholars on opposing sides of ongoing debates are left to quarrel based on competing values, assumptions, and intuitions.

Validating a forced-choice method for eliciting quality-of-reasoning judgments

In this paper we investigate the criterion validity of forced-choice comparisons of the quality of written arguments with normative solutions. Across two studies, novices and experts assessing quality of reasoning through a forced-choice design were both able to choose arguments supporting more accurate solutions-62.2% (SE = 1%) of the time for novices and 74.4% (SE = 1%) for experts-and arguments produced by larger teams-up to 82% of the time for novices and 85% for experts-with high inter-rater reliability, namely 70.58% (95% CI = 1.18) agreement for novices and 80.98% (95% CI = 2.26) for experts. We also explored two methods for increasing efficiency. We found that the number of comparative judgments needed could be substantially reduced with little accuracy loss by leveraging transitivity and producing quality-of-reasoning assessments using an AVL tree method. Moreover, a regression model trained to predict scores based on automatically derived linguistic features of participants' judgments achieved a high correlation with the objective accuracy scores of the arguments in our dataset. Despite the inherent subjectivity involved in evaluating differing quality of reasoning, the forced-choice paradigm allows even novice raters to perform beyond chance and can provide a valid, reliable, and efficient method for producing quality-of-reasoning assessments at scale.

Human and Algorithmic Predictions in Geopolitical Forecasting: Quantifying Uncertainty in Hard-to-Quantify Domains

Research on clinical versus statistical prediction has demonstrated that algorithms make more accurate predictions than humans in many domains. Geopolitical forecasting is an algorithm-unfriendly domain, with hard-to-quantify data and elusive reference classes that make predictive model-building difficult. Furthermore, the stakes can be high, with missed forecasts leading to mass-casualty consequences. For these reasons, geopolitical forecasting is typically done by humans, even though algorithms play important roles. They are essential as aggregators of crowd wisdom, as frameworks to partition human forecasting variance, and as inputs to hybrid forecasting models. Algorithms are extremely important in this domain. We doubt that humans will relinquish control to algorithms anytime soon-nor do we think they should. However, the accuracy of forecasts will greatly improve if humans are aided by algorithms.

AI and the transformation of social science research

Careful bias management and data fidelity are key.

Harm Hypervigilance in Public Reactions to Scientific Evidence

Two preregistered studies from two different platforms with representative U.S. adult samples (N = 1,865) tested the harm-hypervigilance hypothesis in risk assessments of controversial behavioral science. As expected, across six sets of scientific findings, people consistently overestimated others' harmful reactions (medium to large average effect sizes) and underestimated helpful ones, even when incentivized for accuracy. Additional analyses found that (a) harm overestimations were associated with support for censoring science, (b) people who were more offended by scientific findings reported greater difficulty understanding them, and (c) evidence was moderately consistent for an association between more conservative ideology and harm overestimations. These findings are particularly relevant because journals have begun evaluating potential downstream harms of scientific findings. We discuss implications of our work and invite scholars to develop rigorous tests of (a) the social pressures that lead science astray and (b) the actual costs and benefits of publishing or not publishing potentially controversial conclusions.

Insights into the accuracy of social scientists' forecasts of societal change

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists' forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.

From discipline-centered rivalries to solution-centered science: Producing better probability estimates for policy makers

From 2011 to 2015, the U.S. intelligence community sponsored a series of forecasting tournaments that challenged university-based researchers to invent measurably better methods of forecasting political events. Our group, the Good Judgment Project, won these tournaments by balancing the collaboration and competition of members across disciplines. At the outset, psychologists were ahead of economists in identifying individual differences in forecasting skill and developing methods of debiasing forecasts, whereas economists were ahead of psychologists in designing simple market mechanisms that distilled predictive signals from noisy individual-level data. Working closely with statisticians, psychologists eventually beat the markets by producing better probability estimates that funneled top forecasters into elite teams and aggregated their judgments using a log-odds formula tuned to the diversity of the forecasters. Our research group performed best when team members strove to get as much as possible from their home disciplines, but acknowledged their limitations and welcomed help from outsiders. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Correcting Judgment Correctives in National Security Intelligence

Intelligence analysts, like other professionals, form norms that define standards of tradecraft excellence. These norms, however, have evolved in an idiosyncratic manner that reflects the influence of prominent insiders who had keen psychological insights but little appreciation for how to translate those insights into testable hypotheses. The net result is that the prevailing tradecraft norms of best practice are only loosely grounded in the science of judgment and decision-making. The "common sense" of prestigious opinion leaders inside the intelligence community has pre-empted systematic validity testing of the training techniques and judgment aids endorsed by those opinion leaders. Drawing on the scientific literature, we advance hypotheses about how current best practices could well be reducing rather than increasing the quality of analytic products. One set of hypotheses pertain to the failure of tradecraft training to recognize the most basic threat to accuracy: measurement error in the interpretation of the same data and in the communication of interpretations. Another set of hypotheses focuses on the insensitivity of tradecraft training to the risk that issuing broad-brush, one-directional warnings against bias (e.g., over-confidence) will be less likely to encourage self-critical, deliberative cognition than simple response-threshold shifting that yields the mirror-image bias (e.g., under-confidence). Given the magnitude of the consequences of better and worse intelligence analysis flowing to policy-makers, we see a compelling case for greater funding of efforts to test what actually works.

Interfacing With Faces: Perceptual Humanization and Dehumanization

This article links the visual perception of faces and social behavior. We argue that the ways in which people visually encode others’ faces—a rapid-fire perceptual categorization—can result in either humanizing or dehumanizing modes of perception. Our model suggests that these perceptual pathways channel subsequent social inferences and behavior. We focus on the construct of perceptual dehumanization, which involves a shift from configural to featural processing of human faces and, in turn, enables the infliction of harm, such as harsh punishments. We discuss visual attention as an antecedent of perceptual modes and consequent modes of social behavior and speculate about the functions of humanization and dehumanization in sustaining macro-level social structures.

Bringing probability judgments into policy debates via forecasting tournaments

Political debates often suffer from vague-verbiage predictions that make it difficult to assess accuracy and improve policy. A tournament sponsored by the U.S. intelligence community revealed ways in which forecasters can better use probability estimates to make predictions-even for seemingly "unique" events-and showed that tournaments are a useful tool for generating knowledge. Drawing on the literature about the effects of accountability, the authors suggest that tournaments may hold even greater potential as tools for depolarizing political debates and resolving policy disputes.

An Alternative Metaphor in the Study of Judgment and Choice: People as Politicians

Researchers who study judgment and choice have assumed it is useful to think of people as either intuitive psychologists or economists who strive either to understand the people around them or to maximize subjective expected utility. This article proposes that it is also useful to think of people as intuitive politicians who strive to please the diverse constituencies to whom they feel accountable. The article specifies two hard-core assumptions of a political research program on judgment and choice and a testable middle-range theory (the social contingency model). The article also explores both the descriptive and normative implications of shifting from the psychologist and economist metaphors to the politician metaphor. At a descriptive level, the political research program highlights a variety of social and institutional variables that moderate (either attenuate or exacerbate) response tendencies that have been labelled as errors or biases. At a normative level, the political research program highlights an alternative set of criteria for labelling effects as errors or biases. Response tendencies that look flawed from one metaphorical perspective often look quite reasonable from another. The politician metaphor suggests more normatively generous ways of viewing several effects, including the fundamental attribution error, the dilution effect, ambiguity aversion, the attraction effect and the status quo effect.

Biculturalism

Growing numbers of people are being exposed to a second culture, yet the process by which individuals absorb a cultural identity and the role played by second-culture exposure in shaping sociocognitive skills have received little theoretical attention. This article begins to fill these knowledge gaps by delineating the factors that affect the adoption of specific acculturation strategies and focusing on the power of secondculture exposure to stimulate integratively complex cognitions that give people the flexibility to shift rapidly from one cultural meaning system to another. We propose a model, influenced by prior work on value pluralism and accountability, which outlines the underlying mechanisms that determine acculturation choice and that produce both individual difference and situational variation in integrative complexity of social functioning. Implications for expatriate performance are discussed.

Coping with Accountability Cross-Pressures: Low-Effort Evasive Tactics and High-Effort Quests for Complex Compromises

The current study explores two classes of strategies of coping with accountability: low-cognitive-effort decision-evasion tactics (buckpassing, procrastination, and exiting the situation) and high-cognitive-effort attempts to craft integratively complex compromises among conflicting perspectives. Some participants read weak arguments on one side of the free trade issue and strong arguments on the other side, and some participants read strong arguments for both the pro-and anti-free trade positions. They then expected their own views to be anonymous or expected to justify those views to a pro-free trade audience or to both a pro-and an anti-free trade audience. Participants were most integratively complex when they read strong arguments from each side and were accountable to conflicting constituencies (maximum intrapsychic and interpersonal conflict). Participants also relied on low-effort decision-evasion tactics to escape accountability and were willing to use escape strategies demanding relatively more time and energy to avoid accountability to contradictory constituencies.

Accountability and Close-Call Counterfactuals: The Loser Who Nearly Won and the Winner Who Nearly Lost

This article links recent work on assimilative and contrastive counterfactual thinking with research on the impact of accountability on judgment and choice. Relative to participants who felt accountable solely for bottom-line performance outcomes, participants who were accountable for their decision-making process (a) had more pronounced differential reactions to clearly winning versus (winning but) nearly losing and to clearly losing versus (losing but) nearly winning; (b) were less satisfied with the quality of their decisions when they nearly lost and more satisfied with the quality of their decisions when they nearly won; and (c) invested less money into investments that nearly failed and more money into investments that nearly succeeded. This pattern is consistent with the hypothesis that process accountability amplified assimilative counterfactual thinking, whereas outcome accountability attenuated it. The evidence underscores the power of contextual features of the decision-making environment to shape key cognitive and affective consequences of upward and downward counterfactual comparisons.

Sober Second Thought: The Effects of Accountability, Anger, and Authoritarianism on Attributions of Responsibility

This experiment explored the joint impact of accountability, anger, and authoritarianism on attributions of responsibility. Participants were either accountable or anonymous while watching an anger-priming or a neutral-emotion-priming video clip. In an ostensibly separate study, participants also were either accountable or anonymous while determining responsibility and punishment in fictional tort cases. As predicted, priming anger both simplified cognitive processing(i.e., reduced the number of cues used in making judgments) and amplified the carryover of self-reported anger to punitive attributions and actual punishment. By contrast, accountability increased the complexity of the judgment process and attenuated the carryover of anger to attributions and punishment. These results generalized across four replication cases that varied in story content; degree of defendant intentionality; and target, type, and severity of harm.

Improving Intelligence Analysis With Decision Science

Intelligence analysis plays a vital role in policy decision making. Key functions of intelligence analysis include accurately forecasting significant events, appropriately characterizing the uncertainties inherent in such forecasts, and effectively communicating those probabilistic forecasts to stakeholders. We review decision research on probabilistic forecasting and uncertainty communication, drawing attention to findings that could be used to reform intelligence processes and contribute to more effective intelligence oversight. We recommend that the intelligence community (IC) regularly and quantitatively monitor its forecasting accuracy to better understand how well it is achieving its functions. We also recommend that the IC use decision science to improve these functions (namely, forecasting and communication of intelligence estimates made under conditions of uncertainty). In the case of forecasting, decision research offers suggestions for improvement that involve interventions on data (e.g., transforming forecasts to debias them) and behavior (e.g., via selection, training, and effective team structuring). In the case of uncertainty communication, the literature suggests that current intelligence procedures, which emphasize the use of verbal probabilities, are ineffective. The IC should, therefore, leverage research that points to ways in which verbal probability use may be improved as well as exploring the use of numerical probabilities wherever feasible.

Debunking the Myth of Value-Neutral Virginity: Toward Truth in Scientific Advertising

The scientific community often portrays science as a value-neutral enterprise that crisply demarcates facts from personal value judgments. We argue that this depiction is unrealistic and important to correct because science serves an important knowledge generation function in all modern societies. Policymakers often turn to scientists for sound advice, and it is important for the wellbeing of societies that science delivers. Nevertheless, scientists are human beings and human beings find it difficult to separate the epistemic functions of their judgments (accuracy) from the social-economic functions (from career advancement to promoting moral-political causes that “feel self-evidently right”). Drawing on a pluralistic social functionalist framework that identifies five functionalist mindsets—people as intuitive scientists, economists, politicians, prosecutors, and theologians—we consider how these mindsets are likely to be expressed in the conduct of scientists. We also explore how the context of policymaker advising is likely to activate or de-activate scientists’ social functionalist mindsets. For instance, opportunities to advise policymakers can tempt scientists to promote their ideological beliefs and values, even if advising also brings with it additional accountability pressures. We end prescriptively with an appeal to scientists to be more circumspect in characterizing their objectivity and honesty and to reject idealized representations of scientific behavior that inaccurately portray scientists as value-neutral virgins.

Balancing Fairness and Efficiency: The Impact of Identity-Blind and Identity-Conscious Accountability on Applicant Screening

This study compared two forms of accountability that can be used to promote diversity and fairness in personnel selections: identity-conscious accountability (holding decision makers accountable for which groups are selected) versus identity-blind accountability (holding decision makers accountable for making fair selections). In a simulated application screening process, undergraduate participants (majority female) sorted applicants under conditions of identity-conscious accountability, identity-blind accountability, or no accountability for an applicant pool in which white males either did or did not have a human capital advantage. Under identity-conscious accountability, participants exhibited pro-female and pro-minority bias, particularly in the white-male-advantage applicant pool. Under identity-blind accountability, participants exhibited no biases and candidate qualifications dominated interview recommendations. Participants exhibited greater resentment toward management under identity-conscious accountability.

Rational Versus Irrational Prejudices: How Problematic Is the Ideological Lopsidedness of Social Psychology?

The Inbar and Lammers (2012, this issue) survey revealed potentially disturbing pockets of ideological intolerance among social psychologists. Their findings raise a mix of procedural-justice questions bearing on fair treatment of colleagues and epistemological questions bearing on the opportunity costs of research programs that were either never conceived or smothered in infancy. An appropriately self-critical disciplinary response is to conduct identity-substitution thought experiments that explore (a) how we would collectively react to differential treatment in counterfactual worlds in which minority-majority faction roles were reversed, (b) how ideological bias may have suppressed research with the potential to undermine liberal policy positions (e.g., affirmative action, income redistribution, "dove-ish" security policies), and (c) how ideological bias may have led to attaching labels to conservative policy positions that moderates and conservatives consider tendentious (e.g., system justification, symbolic racism).

Brutality Under Cover of Ambiguity

Five studies tested four hypotheses on the drivers of punitive judgments. Study 1 showed that people imposed covertly retributivist physical punishments on extreme norm violators when they could plausibly deny that is what they were doing (attributional ambiguity). Studies 2 and 3 showed that covert retributivism could be suppressed by subtle accountability manipulations that cue people to the possibility that they might be under scrutiny. Studies 4 and 5 showed how covert retributivism can become self-sustaining by biasing the lessons people learn from experience. Covert retributivists did not scale back punitiveness in response to feedback that the justice system makes false-conviction errors but they did ramp up punitiveness in response to feedback that the system makes false-acquittal errors. Taken together, the results underscore the paradoxical nature of covert retributivism: It is easily activated by plausible deniability and persistent in the face of false-conviction feedback but also easily deactivated by minimalist forms of accountability.

Forecasting Tournaments

Forecasting tournaments are level-playing-field competitions that reveal which individuals, teams, or algorithms generate more accurate probability estimates on which topics. This article describes a massive geopolitical tournament that tested clashing views on the feasibility of improving judgmental accuracy and on the best methods of doing so. The tournament’s winner, the Good Judgment Project, outperformed the simple average of the crowd by (a) designing new forms of cognitive-debiasing training, (b) incentivizing rigorous thinking in teams and prediction markets, (c) skimming top talent into elite collaborative teams of “super forecasters,” and (d) fine-tuning aggregation algorithms for distilling greater wisdom from crowds. Tournaments have the potential to open closed minds and increase assertion-to-evidence ratios in polarized scientific and policy debates.

Psychological strategies for winning a geopolitical forecasting tournament

Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracking are psychological interventions that dramatically increased the accuracy of forecasts. Statistical algorithms (reported elsewhere) improved the accuracy of the aggregation. Putting both statistics and psychology to work produced the best forecasts 2 years in a row.

Situated Social Identities Constrain Morally Defensible Choices: Commentary on Bennis, Medin, & Bartels (2010)

Conceptual distinctions that loom large to philosophers-such as the distinction between utilitarian and deontic decision norms-may be far less salient to most other mortals. Building on an intuitive-politician model of judgment and choice and on the empirical work reported by Bennis, Medin, and Bartels (2010, this issue), we argue that the overriding goal of most decision makers in the paradigms under scrutiny is to offer judgments that are readily defensible and that reinforce their social identities as both cognitively flexible (responsive to evidence and cost-benefit considerations) and morally principled (prepared to defend sacred values and censure those who do not). People are best classified neither as utilitarians nor Kantians but rather as pragmatic social beings embedded in complex cultural-political systems.