What Can Experimental Studies of Bias Tell Us About Real-World Group Disparities?

The RepVig framework for designing use-case specific representative vignettes and evaluating triage accuracy of laypeople and symptom assessment applications

Most studies evaluating symptom-assessment applications (SAAs) rely on a common set of case vignettes that are authored by clinicians and devoid of context, which may be representative of clinical settings but not of situations where patients use SAAs. Assuming the use case of self-triage, we used representative design principles to sample case vignettes from online platforms where patients describe their symptoms to obtain professional advice and compared triage performance of laypeople, SAAs (e.g., WebMD or NHS 111), and Large Language Models (LLMs, e.g., GPT-4 or Claude) on representative versus standard vignettes. We found performance differences in all three groups depending on vignette type: When using representative vignettes, accuracy was higher (OR = 1.52 to 2.00, p < .001 to .03 in binary decisions, i.e., correct or incorrect), safety was higher (OR = 1.81 to 3.41, p < .001 to .002 in binary decisions, i.e., safe or unsafe), and the inclination to overtriage was also higher (OR = 1.80 to 2.66, p < .001 to p = .035 in binary decisions, overtriage or undertriage error). Additionally, we found changed rankings of best-performing SAAs and LLMs. Based on these results, we argue that our representative vignette sampling approach (that we call the RepVig Framework) should replace the practice of using a fixed vignette set as standard for SAA evaluation studies.

Insights into the accuracy of social scientists' forecasts of societal change

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists' forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.

Eristic reasoning: Adaptation to extreme uncertainty

Heuristics (shortcut solution rules) can help adaptation to uncertainty by leading to sufficiently accurate decisions with little information. However, heuristics would fail under extreme uncertainty where information is so scarce that any heuristic would be highly misleading for accuracy-seeking. Thus, under very high levels of uncertainty, decision-makers rely on heuristics to no avail. We posit that eristic reasoning (i.e., self-serving inferences for hedonic pursuits), rather than heuristic reasoning, is adaptive when uncertainty is extreme, as eristic reasoning produces instant hedonic gratifications helpful for coping. Eristic reasoning aims at hedonic gains (e.g., relief from the anxiety of uncertainty) that can be pursued by self-serving inferences. As such, eristic reasoning does not require any information about the environment as it instead gets cues introspectively from bodily signals informing what the organism hedonically needs as shaped by individual differences. We explain how decision-makers can benefit from heuristic vs. eristic reasoning under different levels of uncertainty. As a result, by integrating the outputs of formerly published empirical research and our conceptual discussions pertaining to eristic reasoning, we conceptually criticize the fast-and-frugal heuristics approach, which implies that heuristics are the only means of adapting to uncertainty.

Racial Bias in Police Traffic Stops: White Residents' County-Level Prejudice and Stereotypes Are Related to Disproportionate Stopping of Black Drivers

Racial disparities in policing are well documented, but the reasons for such disparities are often debated. In the current research, we weighed in on this debate using a regional-level bias framework: We investigated the link between racial disparities in police traffic stops and regional-level racial bias, employing data from more than 130 million police traffic stops in 1,413 U.S. counties and county-level measures of racial bias from more than 2 million online respondents. Compared with their population share in county demographics, Black drivers were stopped at disproportionate rates in the majority of counties. Crucially, disproportionate stopping of Black drivers was higher in counties with higher levels of racial prejudice by White residents (rs = .07-.36). Furthermore, county-level aggregates of White people's threat-related stereotypes were less consistent in predicting disproportionate stopping (rs = .00-.19). These observed relationships between regional-level bias and racial disparities in policing highlight the importance of the context in which police operate.

Beyond Experiments

It is often claimed that only experiments can support strong causal inferences and therefore they should be privileged in the behavioral sciences. We disagree. Overvaluing experiments results in their overuse both by researchers and decision makers and in an underappreciation of their shortcomings. Neglect of other methods often follows. Experiments can suggest whether X causes Y in a specific experimental setting; however, they often fail to elucidate either the mechanisms responsible for an effect or the strength of an effect in everyday natural settings. In this article, we consider two overarching issues. First, experiments have important limitations. We highlight problems with external, construct, statistical-conclusion, and internal validity; replicability; and conceptual issues associated with simple X causes Y thinking. Second, quasi-experimental and nonexperimental methods are absolutely essential. As well as themselves estimating causal effects, these other methods can provide information and understanding that goes beyond that provided by experiments. A research program progresses best when experiments are not treated as privileged but instead are combined with these other methods.