Artificial Collective Intelligence Engineering: A Survey of Concepts and Perspectives

Collectiveness is an important property of many systems-both natural and artificial. By exploiting a large number of individuals, it is often possible to produce effects that go far beyond the capabilities of the smartest individuals or even to produce intelligent collective behavior out of not-so-intelligent individuals. Indeed, collective intelligence, namely, the capability of a group to act collectively in a seemingly intelligent way, is increasingly often a design goal of engineered computational systems-motivated by recent technoscientific trends like the Internet of Things, swarm robotics, and crowd computing, to name only a few. For several years, the collective intelligence observed in natural and artificial systems has served as a source of inspiration for engineering ideas, models, and mechanisms. Today, artificial and computational collective intelligence are recognized research topics, spanning various techniques, kinds of target systems, and application domains. However, there is still a lot of fragmentation in the research panorama of the topic within computer science, and the verticality of most communities and contributions makes it difficult to extract the core underlying ideas and frames of reference. The challenge is to identify, place in a common structure, and ultimately connect the different areas and methods addressing intelligent collectives. To address this gap, this article considers a set of broad scoping questions providing a map of collective intelligence research, mostly by the point of view of computer scientists and engineers. Accordingly, it covers preliminary notions, fundamental concepts, and the main research perspectives, identifying opportunities and challenges for researchers on artificial and computational collective intelligence engineering.

Majority rule can help solve difficult tasks even when confident members opt out to serve individual interests

When sharing a common goal, confident and competent members are often motivated to contribute to the group, boosting its decision performance. However, it is unclear whether this process remains effective when members can opt in or out of group decisions and prioritize individual interests. Our laboratory experiment (n = 63) and cognitive modeling showed that at the individual level, confidence, competence, and a preference for risk motivated participants' opt-out decisions. We then analyzed the group-level accuracy of majority decisions by creating many virtual groups of 25 members resampled from the 63 participants in the experiment. Whereas the majority decisions by voters who preferred to participate in group decision making were inferior to individual decisions by loners who opted out in an easy task, this was reversed in a difficult task. Bootstrap-simulation analyses decomposed these outcomes into the effects of a decrease in group size and a decrease in voters' accuracy accruing from the opt-in/out mechanism, demonstrating how these effects interacted with task difficulty. Our results suggest that the majority rule still works to tackle challenging problems even when individual interests are emphasized over collective performance, playing a functional as well as a democratic role in consensus decision making under uncertainty.

The network science of collective intelligence

In the last few years, breakthroughs in computational and experimental techniques have produced several key discoveries in the science of networks and human collective intelligence. This review presents the latest scientific findings from two key fields of research: collective problem-solving and the wisdom of the crowd. I demonstrate the core theoretical tensions separating these research traditions and show how recent findings offer a new synthesis for understanding how network dynamics alter collective intelligence, both positively and negatively. I conclude by highlighting current theoretical problems at the forefront of research on networked collective intelligence, as well as vital public policy challenges that require new research efforts.

The motor Wisdom of the Crowd

Wisdom of the Crowd is the aggregation of many individual estimates to obtain a better collective one. Because of its enormous social potential, this effect has been thoroughly investigated, but predominantly on tasks that involve rational thinking (such as estimating a number). Here we tested this effect in the context of drawing geometrical shapes, which still enacts cognitive processes but mainly involves visuomotor control. We asked more than 700 school students to trace five patterns shown on a touchscreen and then aggregated their individual trajectories to improve the match with the original pattern. Our results show the characteristics of the strongest examples of Wisdom of the Crowd. First, the aggregate trajectory can be up to 5 times more accurate than the individual ones. Second, this great improvement requires aggregating trajectories from different individuals (rather than trials from the same individual). Third, the aggregate trajectory outperforms more than 99% of individual trajectories. Fourth, while older individuals outperform younger ones, a crowd of young individuals outperforms the average older one. These results demonstrate for the first time Wisdom of the Crowd in the realm of motor control, opening the door to further studies of human and also animal behavioural trajectories and their mechanistic underpinnings.

The potential for effective reasoning guides children's preference for small group discussion over crowdsourcing

Communication between social learners can make a group collectively "wiser" than any individual, but conformist tendencies can also distort collective judgment. We asked whether intuitions about when communication is likely to improve or distort collective judgment could allow social learners to take advantage of the benefits of communication while minimizing the risks. In three experiments (n = 360), 7- to 10-year old children and adults decided whether to refer a question to a small group for discussion or "crowdsource" independent judgments from individual advisors. For problems affording the kind of 'demonstrative' reasoning that allows a group member to reliably correct errors made by even a majority, all ages preferred to consult the discussion group, even compared to a crowd ten times as large-consistent with past research suggesting that discussion groups regularly outperform even their best members for reasoning problems. In contrast, we observed a consistent developmental shift towards crowdsourcing independent judgments when reasoning by itself was insufficient to conclusively answer a question. Results suggest sophisticated intuitions about the nature of social influence and collective intelligence may guide our social learning strategies from early in development.

E-proctored exams during the COVID-19 pandemic: A close understanding

Researchers have focused on evaluating and exploring the online examination experience during the COVID-19 pandemic. However, understanding the perceptions of using an e-proctoring tool within the online examination experience is still limited. This study explores the first unique experience for students' attitudes and concerns using an e-proctoring tool in their final exams during the COVID-19 pandemic. It also highlights the e-tools' impact on students' performances to guide educational institutions towards appropriate practices going forward, especially as the pandemic is expected to have far-reaching consequences. A mixed-methods analysis was used to examine heterogeneous sources of data including self-reported data and officially documented data. The data was analyzed by a qualitative analysis of the focus group and quantitative analyses of the survey questions and exam attempts. In June 2020, students participated in a focus group to elaborate on their attitudes and concerns pertaining to their e-proctoring experience. Based on the preliminary outcomes, a survey was developed and distributed to a purposive sample (n = 106) of students from information technology majors who had taken at least one e-proctored exam during the COVID-19 pandemic. Finally, 21 online exams with 815 total attempts were analyzed to assess how well students performed under an e-proctored test. The study's findings shed light on students' perceptions of their e-proctoring experience, including their predominant concerns over privacy and various environmental and psychological factors. The research also highlights challenges in implementing the e-proctoring tool as well as its impact on students' performance.

Multi-model and network inference based on ensemble estimates: avoiding the madness of crowds

Recent progress in theoretical systems biology, applied mathematics and computational statistics allows us to compare the performance of different candidate models at describing a particular biological system quantitatively. Model selection has been applied with great success to problems where a small number-typically less than 10-of models are compared, but recent studies have started to consider thousands and even millions of candidate models. Often, however, we are left with sets of models that are compatible with the data, and then we can use ensembles of models to make predictions. These ensembles can have very desirable characteristics, but as I show here are not guaranteed to improve on individual estimators or predictors. I will show in the cases of model selection and network inference when we can trust ensembles, and when we should be cautious. The analyses suggest that the careful construction of an ensemble-choosing good predictors-is of paramount importance, more than had perhaps been realized before: merely adding different methods does not suffice. The success of ensemble network inference methods is also shown to rest on their ability to suppress false-positive results. A Jupyter notebook which allows carrying out an assessment of ensemble estimators is provided.

Counteracting estimation bias and social influence to improve the wisdom of crowds

Aggregating multiple non-expert opinions into a collective estimate can improve accuracy across many contexts. However, two sources of error can diminish collective wisdom: individual estimation biases and information sharing between individuals. Here, we measure individual biases and social influence rules in multiple experiments involving hundreds of individuals performing a classic numerosity estimation task. We first investigate how existing aggregation methods, such as calculating the arithmetic mean or the median, are influenced by these sources of error. We show that the mean tends to overestimate, and the median underestimate, the true value for a wide range of numerosities. Quantifying estimation bias, and mapping individual bias to collective bias, allows us to develop and validate three new aggregation measures that effectively counter sources of collective estimation error. In addition, we present results from a further experiment that quantifies the social influence rules that individuals employ when incorporating personal estimates with social information. We show that the corrected mean is remarkably robust to social influence, retaining high accuracy in the presence or absence of social influence, across numerosities and across different methods for averaging social information. Using knowledge of estimation biases and social influence rules may therefore be an inexpensive and general strategy to improve the wisdom of crowds.

Deep attention networks reveal the rules of collective motion in zebrafish

A variety of simple models has been proposed to understand the collective motion of animals. These models can be insightful but may lack important elements necessary to predict the motion of each individual in the collective. Adding more detail increases predictability but can make models too complex to be insightful. Here we report that deep attention networks can obtain a model of collective behavior that is simultaneously predictive and insightful thanks to an organization in modules. When using simulated trajectories, the model recovers the ground-truth interaction rule used to generate them, as well as the number of interacting neighbours. For experimental trajectories of large groups of 60-100 zebrafish, Danio rerio, the model obtains that interactions between pairs can approximately be described as repulsive, attractive or as alignment, but only when moving slowly. At high velocities, interactions correspond only to alignment or alignment mixed with repulsion at close distances. The model also shows that each zebrafish decides where to move by aggregating information from the group as a weighted average over neighbours. Weights are higher for neighbours that are close, in a collision path or moving faster in frontal and lateral locations. The network also extracts that the number of interacting individuals is dynamical and typically in the range 8–22, with 1–10 more important ones. Our results suggest that each animal decides by dynamically selecting information from the collective.

Simple models have traditionally been very successful, because they usually provide more insight than complicated models. This is particularly true in physics, where simple models can often give highly precise quantitative predictions. However, biology is fundamentally complex and thus it is difficult to find simple models that give precise predictions. To create models that are both precise and insightful, we propose to harness the power of deep neural networks but to confine them into modules with a low number of inputs and outputs. We trained one such model to predict the future turning side of a fish in a collective. By plotting the different modules we obtain insight about how fish interact and how they aggregate information from different neighbours. This aggregation is dynamical and shows that fish can interact with approximately 20 neighbours but can also focus on fewer neighbours, down to 1-2, when some move at higher speed in front or to the sides, are very close or are in a collision path.

Individuals fail to reap the collective benefits of diversity because of over-reliance on personal information

Collective intelligence refers to the ability of groups to outperform individuals in solving cognitive tasks. Although numerous studies have demonstrated this effect, the mechanisms underlying collective intelligence remain poorly understood. Here, we investigate diversity in cue beliefs as a mechanism potentially promoting collective intelligence. In our experimental study, human groups observed a sequence of cartoon characters, and classified each character as a cooperator or defector based on informative and uninformative cues. Participants first made an individual decision. They then received social information consisting of their group members' decisions before making a second decision. Additionally, individuals reported their beliefs about the cues. Our results showed that individuals made better decisions after observing the decisions of others. Interestingly, individuals developed different cue beliefs, including many wrong ones, despite receiving identical information. Diversity in cue beliefs, however, did not predict collective improvement. Using simulations, we found that diverse collectives did provide better social information, but that individuals failed to reap those benefits because they relied too much on personal information. Our results highlight the potential of belief diversity for promoting collective intelligence, but suggest that this potential often remains unexploited because of over-reliance on personal information.

Crowd wisdom enhanced by costly signaling in a virtual rating system

Costly signaling theory from ecology posits that signals will be more honest and thus information will be accurately communicated when signaling carries a nontrivial cost. Our study combines this concept from behavioral ecology with methods of computational social science to show how costly signaling can improve crowd wisdom in human, online rating systems. Specifically, we endowed a rating widget with virtual friction to increase the time cost for reporting extreme scores. Even without any conflicts of interests or incentives to cheat, costly signaling helped obtain reliable crowd estimates of quality. Our results have implications for the ubiquitous solicitation of evaluations in e-commerce, and the approach can be generalized and tested in a variety of large-scale online communication systems.

Costly signaling theory was developed in both economics and biology and has been used to explain a wide range of phenomena. However, the theory’s prediction that signal cost can enforce information quality in the design of new communication systems has never been put to an empirical test. Here we show that imposing time costs on reporting extreme scores can improve crowd wisdom in a previously cost-free rating system. We developed an online game where individuals interacted repeatedly with simulated services and rated them for satisfaction. We associated ratings with differential time costs by endowing the graphical user interface that solicited ratings from the users with “physics,” including an initial (default) slider position and friction. When ratings were not associated with differential cost (all scores from 0 to 100 could be given by an equally low-cost click on the screen), scores correlated only weakly with objective service quality. However, introducing differential time costs, proportional to the deviation from the mean score, improved correlations between subjective rating scores and objective service performance and lowered the sample size required for obtaining reliable, averaged crowd estimates. Boosting time costs for reporting extreme scores further facilitated the detection of top performances. Thus, human collective online behavior, which is typically cost-free, can be made more informative by applying costly signaling via the virtual physics of rating devices.

Quorums enable optimal pooling of independent judgements in biological systems

Collective decision-making is ubiquitous, and majority-voting and the Condorcet Jury Theorem pervade thinking about collective decision-making. Thus, it is typically assumed that majority-voting is the best possible decision mechanism, and that scenarios exist where individually-weak decision-makers should not pool information. Condorcet and its applications implicitly assume that only one kind of error can be made, yet signal detection theory shows two kinds of errors exist, 'false positives' and 'false negatives'. We apply signal detection theory to collective decision-making to show that majority voting is frequently sub-optimal, and can be optimally replaced by quorum decision-making. While quorums have been proposed to resolve within-group conflicts, or manage speed-accuracy trade-offs, our analysis applies to groups with aligned interests undertaking single-shot decisions. Our results help explain the ubiquity of quorum decision-making in nature, relate the use of sub- and super-majority quorums to decision ecology, and may inform the design of artificial decision-making systems.

Editorial note

This article has been through an editorial process in which the authors decide how to respond to the issues raised during peer review. The Reviewing Editor's assessment is that all the issues have been addressed (see decision letter).

Social learning strategies regulate the wisdom and madness of interactive crowds

Why groups of individuals sometimes exhibit collective 'wisdom' and other times maladaptive 'herding' is an enduring conundrum. Here we show that this apparent conflict is regulated by the social learning strategies deployed. We examined the patterns of human social learning through an interactive online experiment with 699 participants, varying both task uncertainty and group size, then used hierarchical Bayesian model fitting to identify the individual learning strategies exhibited by participants. Challenging tasks elicit greater conformity among individuals, with rates of copying increasing with group size, leading to high probabilities of herding among large groups confronted with uncertainty. Conversely, the reduced social learning of small groups, and the greater probability that social information would be accurate for less-challenging tasks, generated 'wisdom of the crowd' effects in other circumstances. Our model-based approach provides evidence that the likelihood of collective intelligence versus herding can be predicted, resolving a long-standing puzzle in the literature.

Adolescents show collective intelligence which can be driven by a geometric mean rule of thumb

How effective groups are in making decisions is a long-standing question in studying human and animal behaviour. Despite the limited social and cognitive abilities of younger people, skills which are often required for collective intelligence, studies of group performance have been limited to adults. Using a simple task of estimating the number of sweets in jars, we show in two experiments that adolescents at least as young as 11 years old improve their estimation accuracy after a period of group discussion, demonstrating collective intelligence. Although this effect was robust to the overall distribution of initial estimates, when the task generated positively skewed estimates, the geometric mean of initial estimates gave the best fit to the data compared to other tested aggregation rules. A geometric mean heuristic in consensus decision making is also likely to apply to adults, as it provides a robust and well-performing rule for aggregating different opinions. The geometric mean rule is likely to be based on an intuitive logarithmic-like number representation, and our study suggests that this mental number scaling may be beneficial in collective decisions.