The Strengths and Weaknesses of Crowds to Address Global Problems

Global climate change, the COVID-19 pandemic, and the spread of misinformation on social media are just a handful of highly consequential problems affecting society. We argue that the rough contours of many societal problems can be framed within a "wisdom of crowds" perspective. Such a framing allows researchers to recast complex problems within a simple conceptual framework and leverage known results on crowd wisdom. To this end, we present a simple "toy" model of the strengths and weaknesses of crowd wisdom that easily maps to many societal problems. Our model treats the judgments of individuals as random draws from a distribution intended to represent a heterogeneous population. We use a weighted mean of these individuals to represent the crowd's collective judgment. Using this setup, we show that subgroups have the potential to produce substantively different judgments and we investigate their effect on a crowd's ability to generate accurate judgments about societal problems. We argue that future work on societal problems can benefit from more sophisticated, domain-specific theory and models based on the wisdom of crowds.

Group Structure and Information Distribution on the Emergence of Collective Intelligence

More and more decision-making problems are being solved by groups. Collective intelligence is the ability of groups to perform well when solving complex problems. Thus, it is important to encourage collective intelligence to emerge from groups. In this study, we explore how two critical characteristics of groups, that is, group structure and individual knowledge in groups, influence the emergence of collective intelligence. To do this, we propose a measure for group structure using the collaboration network of a group and a measure for the distribution of individual knowledge in groups. Group structure is measured based on the intensities of links and whether the network is hierarchical or flat. The distribution of individual knowledge is measured from the perspective of whether group information is shared or unique. Social interactions among group members and individual changes in opinion are modeled based on a simulation technique. We find that unbalanced information distribution undermines group performance, whereas group structure can modify the effect of information distribution. We also find that groups with broadly distributed knowledge are good at solving complex problems.

Funding: This work was supported by the National Natural Science Foundation of China [Grants 72171158, 71771156 and 71971145].

Collectives and Epistemic Rationality

Consideration of collectives raises important questions about human rationality. This has long been known for questions about preferences, but it holds also with respect to beliefs. For one, there are contexts (such as voting) where we might care as much, or more, about the rationality of a collective than the rationality of the individuals it comprises. Here, a given standard may yield competing assessments at the individual and the collective level, thus giving rise to important normative questions. At the same time, seemingly rational strategies of individuals may have surprising consequences, or even fail, when exercised by individuals within collectives. This paper will illustrate these considerations with examples, provide an overview of different formal frameworks for understanding and assessing the beliefs of collectives, and it will illustrate how such frameworks can combine with simulations in order to elucidate epistemic norms.

Bayesian modeling of human-AI complementarity

With the increase in artificial intelligence in real-world applications, there is interest in building hybrid systems that take both human and machine predictions into account. Previous work has shown the benefits of separately combining the predictions of diverse machine classifiers or groups of people. Using a Bayesian modeling framework, we extend these results by systematically investigating the factors that influence the performance of hybrid combinations of human and machine classifiers while taking into account the unique ways human and algorithmic confidence is expressed.

Artificial intelligence (AI) and machine learning models are being increasingly deployed in real-world applications. In many of these applications, there is strong motivation to develop hybrid systems in which humans and AI algorithms can work together, leveraging their complementary strengths and weaknesses. We develop a Bayesian framework for combining the predictions and different types of confidence scores from humans and machines. The framework allows us to investigate the factors that influence complementarity, where a hybrid combination of human and machine predictions leads to better performance than combinations of human or machine predictions alone. We apply this framework to a large-scale dataset where humans and a variety of convolutional neural networks perform the same challenging image classification task. We show empirically and theoretically that complementarity can be achieved even if the human and machine classifiers perform at different accuracy levels as long as these accuracy differences fall within a bound determined by the latent correlation between human and machine classifier confidence scores. In addition, we demonstrate that hybrid human–machine performance can be improved by differentiating between the errors that humans and machine classifiers make across different class labels. Finally, our results show that eliciting and including human confidence ratings improve hybrid performance in the Bayesian combination model. Our approach is applicable to a wide variety of classification problems involving human and machine algorithms.

Collective Choice May Tell Nothing About Anyone’s Individual Preferences

As has been known for over a century, aggregated preferences of a group may bear little or no similarity to the preference of any single individual, regardless of the aggregation method. Yet, it remains routine to fit or test theories of individual decision making on pooled data, and it remains routine to cast theories of individual decision making at the aggregate level. This mindset may have disastrous policy and business implications. A population of individuals who all satisfy one theory may behave collectively as though they satisfied a competing theory. A collection of individuals satisfying a given theory may collectively satisfy a version of the same theory with qualitatively different scientific or decision analytic implications. Because the resulting artifacts apply at the population level, replications, large samples, and high-quality data can do nothing to detect or repair them.

A survey of human judgement and quantitative forecasting methods

This paper's top-level goal is to provide an overview of research conducted in the many academic domains concerned with forecasting. By providing a summary encompassing these domains, this survey connects them, establishing a common ground for future discussions. To this end, we survey literature on human judgement and quantitative forecasting as well as hybrid methods that involve both humans and algorithmic approaches. The survey starts with key search terms that identified more than 280 publications in the fields of computer science, operations research, risk analysis, decision science, psychology and forecasting. Results show an almost 10-fold increase in the application-focused forecasting literature between the 1990s and the current decade, with a clear rise of quantitative, data-driven forecasting models. Comparative studies of quantitative methods and human judgement show that (1) neither method is universally superior, and (2) the better method varies as a function of factors such as availability, quality, extent and format of data, suggesting that (3) the two approaches can complement each other to yield more accurate and resilient models. We also identify four research thrusts in the human/machine-forecasting literature: (i) the choice of the appropriate quantitative model, (ii) the nature of the interaction between quantitative models and human judgement, (iii) the training and incentivization of human forecasters, and (iv) the combination of multiple forecasts (both algorithmic and human) into one. This review surveys current research in all four areas and argues that future research in the field of human/machine forecasting needs to consider all of them when investigating predictive performance. We also address some of the ethical dilemmas that might arise due to the combination of quantitative models with human judgement.

Enhancing the Wisdom of the Crowd With Cognitive-Process Diversity: The Benefits of Aggregating Intuitive and Analytical Judgments

Drawing on dual-process theory, we suggest that the benefits that arise from combining several quantitative individual judgments will be heightened when these judgments are based on different cognitive processes. We tested this hypothesis in three experimental studies in which participants provided estimates for the dates of different historical events (Study 1, N = 152), made probabilistic forecasts for the outcomes of soccer games (Study 2, N = 98), and estimated the weight of individuals on the basis of a photograph (Study 3, N = 3,695). For each of these tasks, participants were prompted to make judgments relying on an analytical process, on their intuition, or (in a control condition) on no specific instructions. Across all three studies, our results show that an aggregation of intuitive and analytical judgments provides more accurate estimates than any other aggregation procedure and that this advantage increases with the number of aggregated judgments.

A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment

Research has shown that aggregation of independent expert judgments significantly improves the quality of forecasts as compared to individual expert forecasts. This “wisdom of crowds” (WOC) has sparked substantial interest. However, previous studies on strengths and weaknesses of aggregation algorithms have been restricted by limited empirical data and analytical complexity. Based on a comprehensive analysis of existing knowledge on WOC and aggregation algorithms, this paper describes the design and implementation of a static stochastic simulation model to emulate WOC scenarios with a wide range of parameters. The model has been thoroughly evaluated: the assumptions are validated against propositions derived from literature, and the model has a computational representation. The applicability of the model is demonstrated by investigating aggregation algorithm behavior on a detailed level, by assessing aggregation algorithm performance, and by exploring previously undiscovered suppositions on WOC. The simulation model helps expand the understanding of WOC, where previous research was restricted. Additionally, it gives directions for developing aggregation algorithms and contributes to a general understanding of the WOC phenomenon.

Demographically diverse crowds are typically not much wiser than homogeneous crowds

Averaging independent numerical judgments can be more accurate than the average individual judgment. This "wisdom of crowds" effect has been shown with large, diverse samples, but the layperson wishing to take advantage of this may only have access to the opinions of a small, more demographically homogeneous "convenience sample." How wise are homogeneous crowds relative to diverse crowds? In simulations and survey studies, we demonstrate three necessary conditions under which small socially diverse crowds can outperform socially homogeneous crowds: Social identity must predict judgment, the effect of social identity on judgment must be at least moderate in size, and the average estimates of the social groups in question must "bracket" the truth being judged. Seven survey studies suggest that these conditions are rarely met in real judgment tasks. Comparisons between the performances of diverse and homogeneous crowds further confirm that social diversity can make crowds wiser but typically by a very small margin.

The relationship between crowd majority and accuracy for binary decisions

We consider the wisdom of the crowd situation in which individuals make binary decisions, and the majority answer is used as the group decision. Using data sets from nine different domains, we examine the relationship between the size of the majority and the accuracy of the crowd decisions. We find empirically that these calibration curves take many different forms for different domains, and the distribution of majority sizes over decisions in a domain also varies widely. We develop a growth model for inferring and interpreting the calibration curve in a domain, and apply it to the same nine data sets using Bayesian methods. The modeling approach is able to infer important qualitative properties of a domain, such as whether it involves decisions that have ground truths or are inherently uncertain. It is also able to make inferences about important quantitative properties of a domain, such as how quickly the crowd accuracy increases as the size of the majority increases. We discuss potential applications of the measurement model, and the need to develop a psychological account of the variety of calibration curves that evidently exist.