Conventional and frugal methods of estimating COVID-19-related excess deaths and undercount factors

Across the world, the officially reported number of COVID-19 deaths is likely an undercount. Establishing true mortality is key to improving data transparency and strengthening public health systems to tackle future disease outbreaks. In this study, we estimated excess deaths during the COVID-19 pandemic in the Pune region of India. Excess deaths are defined as the number of additional deaths relative to those expected from pre-COVID-19-pandemic trends. We integrated data from: (a) epidemiological modeling using pre-pandemic all-cause mortality data, (b) discrepancies between media-reported death compensation claims and official reported mortality, and (c) the "wisdom of crowds" public surveying. Our results point to an estimated 14,770 excess deaths [95% CI 9820-22,790] in Pune from March 2020 to December 2021, of which 9093 were officially counted as COVID-19 deaths. We further calculated the undercount factor-the ratio of excess deaths to officially reported COVID-19 deaths. Our results point to an estimated undercount factor of 1.6 [95% CI 1.1-2.5]. Besides providing similar conclusions about excess deaths estimates across different methods, our study demonstrates the utility of frugal methods such as the analysis of death compensation claims and the wisdom of crowds in estimating excess mortality.

The nature of anchor-biased estimates and its application to the wisdom of crowds

We propose a method to achieve better wisdom of crowds by utilizing anchoring effects. In this method, people are first asked to make a comparative judgment such as "Is the number of new COVID-19 infections one month later more or less than 10 (or 200,000)?" As in this example, two sufficiently different anchors (e.g., "10" or "200,000") are set in the comparative judgment. After this comparative judgment, people are asked to make their own estimates. These estimates are then aggregated. We hypothesized that the aggregated estimates using this method would be more accurate than those without anchor presentation. To examine the effectiveness of the proposed method, we conducted three studies: a computer simulation and two behavioral experiments (numerical estimation of perceptual stimuli and estimation of new COVID-19 infections by physicians). Through computer simulations, we could identify situations in which the proposed method is effective. Although the proposed method is not always effective (e.g., when a group can make fairly accurate estimations), on average, the proposed method is more likely to achieve better wisdom of crowds. In particular, when a group cannot make accurate estimations (i.e., shows biases such as overestimation or underestimation), the proposed method can achieve better wisdom of crowds. The results of the behavioral experiments were consistent with the computer simulation findings. The proposed method achieved better wisdom of crowds. We discuss new insights into anchoring effects and methods for inducing diverse opinions from group members.

Harnessing the power of collective intelligence in dentistry: a pilot study in Victoria, Australia

BACKGROUND

In many dental settings, diagnosis and treatment planning is the responsibility of a single clinician, and this process is inevitably influenced by the clinician's own heuristics and biases. Our aim was to test whether collective intelligence increases the accuracy of individual diagnoses and treatment plans, and whether such systems have potential to improve patient outcomes in a dental setting.

METHODS

This pilot project was carried out to assess the feasibility of the protocol and appropriateness of the study design. We used a questionnaire survey and pre-post study design in which dental practitioners were involved in the diagnosis and treatment planning of two simulated cases. Participants were provided the opportunity to amend their original diagnosis/treatment decisions after viewing a consensus report made to simulate a collaborative setting.

RESULTS

Around half (55%, n = 17) of the respondents worked in group private practices, however most practitioners (74%, n = 23) did not collaborate when planning treatment. Overall, the average practitioners' self-confidence score in managing different dental disciplines was 7.22 (s.d. 2.20) on a 1-10 scale. Practitioners tended to change their mind after viewing the consensus response, particularly for the complex case compared to the simple case (61.5% vs 38.5%, respectively). Practitioners' confidence ratings were also significantly higher (p < 0.05) after viewing the consensus for complex case.

CONCLUSION

Our pilot study shows that collective intelligence in the form of peers' opinion can lead to modifications in diagnosis and treatment planning by dentists. Our results lay the foundations for larger scale investigations on whether peer collaboration can improve diagnostic accuracy, treatment planning and, ultimately, oral health outcomes.

Does discussion make crowds any wiser?

Does discussion in large groups help or hinder the wisdom of crowds? To give rise to the wisdom of crowds, by which large groups can yield surprisingly accurate answers, aggregation mechanisms such as averaging of opinions or majority voting rely on diversity of opinions, and independence between the voters. Discussion tends to reduce diversity and independence. On the other hand, discussion in small groups has been shown to improve the accuracy of individual answers. To test the effects of discussion in large groups, we gave groups of participants (N = 1958 participants in groups of size ranging from 22 to 212; mean 59) one of three types of problems (demonstrative, factual, ethical) to solve, first individually, and then through discussion. For demonstrative (logical or mathematical) problems, discussion improved individual answers, as well as the answers reached through aggregation. For factual problems, discussion improved individual answers, and either improved or had no effect on the answers reached through aggregation. Our results suggest that, for problems which have a correct answer, discussion in large groups does not detract from the effects of the wisdom of crowds, and tends on the contrary to improve on it.

Bayesian collective learning emerges from heuristic social learning

Researchers across cognitive science, economics, and evolutionary biology have studied the ubiquitous phenomenon of social learning-the use of information about other people's decisions to make your own. Decision-making with the benefit of the accumulated knowledge of a community can result in superior decisions compared to what people can achieve alone. However, groups of people face two coupled challenges in accumulating knowledge to make good decisions: (1) aggregating information and (2) addressing an informational public goods problem known as the exploration-exploitation dilemma. Here, we show how a Bayesian social sampling model can in principle simultaneously optimally aggregate information and nearly optimally solve the exploration-exploitation dilemma. The key idea we explore is that Bayesian rationality at the level of a population can be implemented through a more simplistic heuristic social learning mechanism at the individual level. This simple individual-level behavioral rule in the context of a group of decision-makers functions as a distributed algorithm that tracks a Bayesian posterior in population-level statistics. We test this model using a large-scale dataset from an online financial trading platform.

Does the "surprisingly popular" method yield accurate crowdsourced predictions?

The “surprisingly popular” method (SP) of aggregating individual judgments has shown promise in overcoming a weakness of other crowdsourcing methods—situations in which the majority is incorrect. This method relies on participants’ estimates of other participants’ judgments; when an option is chosen more often than the average metacognitive judgments of that option, it is “surprisingly popular” and is selected by the method. Although SP has been shown to improve group decision making about factual propositions (e.g., state capitals), its application to future outcomes has been limited. In three preregistered studies, we compared SP to other methods of aggregating individual predictions about future events. Study 1 examined predictions of football games, Study 2 examined predictions of the 2018 US midterm elections, and Study 3 examined predictions of basketball games. When applied to judgments made by objectively assessed experts, SP performed slightly better than other aggregation methods. Although there is still more to learn about the conditions under which SP is effective, it shows promise as a means of crowdsourcing predictions of future outcomes.

Wisdom of crowds and collective decision-making in a survival situation with complex information integration

Background

The wisdom of crowds and collective decision-making are important tools for integrating information between individuals, which can exceed the capacity of individual judgments. They are based on different forms of information integration. The wisdom of crowds refers to the aggregation of many independent judgments without deliberation and consensus, while collective decision-making is aggregation with deliberation and consensus. Recent research has shown that collective decision-making outperforms the wisdom of crowds. Additionally, many studies have shown that metacognitive knowledge of subjective confidence is useful for improving aggregation performance. However, because most of these studies have employed relatively simple problems; for example, involving general knowledge and estimating values and quantities of objects, it remains unclear whether their findings can be generalized to real-life situations involving complex information integration. This study explores the performance and process of the wisdom of crowds and collective decision-making by applying the wisdom of crowds with weighted confidence to a survival situation task commonly used in studies of collective decision-making.

Results

The wisdom of crowds and collective decision-making outperformed individual judgment. However, collective decision-making did not outperform the wisdom of crowds. Contrary to previous studies, weighted confidence showed no advantage from comparison between confidence-weighted and non-weighted aggregations; a simulation analysis varying in group size and sensitivity of confidence weighting revealed interaction between group size and sensitivity of confidence weighting. This reveals that it is because of small group size and not the peculiarity of the survival task that results in no advantage of weighted confidence.

Conclusions

The study’s findings suggest that the wisdom of crowds could be applicable to complex problem-solving tasks, and interaction between group size and sensitivity of confidence weighting is important for confidence-weighted aggregation effects.

Collective intelligence in fingerprint analysis

When a fingerprint is located at a crime scene, a human examiner is counted upon to manually compare this print to those stored in a database. Several experiments have now shown that these professional analysts are highly accurate, but not infallible, much like other fields that involve high-stakes decision-making. One method to offset mistakes in these safety-critical domains is to distribute these important decisions to groups of raters who independently assess the same information. This redundancy in the system allows it to continue operating effectively even in the face of rare and random errors. Here, we extend this "wisdom of crowds" approach to fingerprint analysis by comparing the performance of individuals to crowds of professional analysts. We replicate the previous findings that individual experts greatly outperform individual novices, particularly in their false-positive rate, but they do make mistakes. When we pool the decisions of small groups of experts by selecting the decision of the majority, however, their false-positive rate decreases by up to 8% and their false-negative rate decreases by up to 12%. Pooling the decisions of novices results in a similar drop in false negatives, but increases their false-positive rate by up to 11%. Aggregating people's judgements by selecting the majority decision performs better than selecting the decision of the most confident or the most experienced rater. Our results show that combining independent judgements from small groups of fingerprint analysts can improve their performance and prevent these mistakes from entering courts.

Aggregation mechanisms for crowd predictions

When the information of many individuals is pooled, the resulting aggregate often is a good predictor of unknown quantities or facts. This aggregate predictor frequently outperforms the forecasts of experts or even the best individual forecast included in the aggregation process ("wisdom of crowds"). However, an appropriate aggregation mechanism is considered crucial to reaping the benefits of a "wise crowd". Of the many possible ways to aggregate individual forecasts, we compare (uncensored and censored) arithmetic and geometric mean and median, continuous double auction market prices and sealed bid-offer call market prices in a controlled experiment. We use an asymmetric information structure, where participants know different sub-sets of the total information needed to exactly calculate the asset value to be estimated. We find that prices from continuous double auction markets clearly outperform all alternative approaches for aggregating dispersed information and that information lets only the best-informed participants generate excess returns.

The wisdom of crowds in action: Forecasting epidemic diseases with a web-based prediction market system

BACKGROUND

The quest for an effective system capable of monitoring and predicting the trends of epidemic diseases is a critical issue for communities worldwide. With the prevalence of Internet access, more and more researchers today are using data from both search engines and social media to improve the prediction accuracy. In particular, a prediction market system (PMS) exploits the wisdom of crowds on the Internet to effectively accomplish relatively high accuracy.

OBJECTIVE

This study presents the architecture of a PMS and demonstrates the matching mechanism of logarithmic market scoring rules. The system was implemented to predict infectious diseases in Taiwan with the wisdom of crowds in order to improve the accuracy of epidemic forecasting.

METHODS

The PMS architecture contains three design components: database clusters, market engine, and Web applications. The system accumulated knowledge from 126 health professionals for 31 weeks to predict five disease indicators: the confirmed cases of dengue fever, the confirmed cases of severe and complicated influenza, the rate of enterovirus infections, the rate of influenza-like illnesses, and the confirmed cases of severe and complicated enterovirus infection.

RESULTS

Based on the winning ratio, the PMS predicts the trends of three out of five disease indicators more accurately than does the existing system that uses the five-year average values of historical data for the same weeks. In addition, the PMS with the matching mechanism of logarithmic market scoring rules is easy to understand for health professionals and applicable to predict all the five disease indicators.

CONCLUSIONS

The PMS architecture of this study affords organizations and individuals to implement it for various purposes in our society. The system can continuously update the data and improve prediction accuracy in monitoring and forecasting the trends of epidemic diseases. Future researchers could replicate and apply the PMS demonstrated in this study to more infectious diseases and wider geographical areas, especially the under-developed countries across Asia and Africa.

Integration of Social Information by Human Groups

We consider a situation in which individuals search for accurate decisions without direct feedback on their accuracy, but with information about the decisions made by peers in their group. The "wisdom of crowds" hypothesis states that the average judgment of many individuals can give a good estimate of, for example, the outcomes of sporting events and the answers to trivia questions. Two conditions for the application of wisdom of crowds are that estimates should be independent and unbiased. Here, we study how individuals integrate social information when answering trivia questions with answers that range between 0% and 100% (e.g., "What percentage of Americans are left-handed?"). We find that, consistent with the wisdom of crowds hypothesis, average performance improves with group size. However, individuals show a consistent bias to produce estimates that are insufficiently extreme. We find that social information provides significant, albeit small, improvement to group performance. Outliers with answers far from the correct answer move toward the position of the group mean. Given that these outliers also tend to be nearer to 50% than do the answers of other group members, this move creates group polarization away from 50%. By looking at individual performance over different questions we find that some people are more likely to be affected by social influence than others. There is also evidence that people differ in their competence in answering questions, but lack of competence is not significantly correlated with willingness to change guesses. We develop a mathematical model based on these results that postulates a cognitive process in which people first decide whether to take into account peer guesses, and if so, to move in the direction of these guesses. The size of the move is proportional to the distance between their own guess and the average guess of the group. This model closely approximates the distribution of guess movements and shows how outlying incorrect opinions can be systematically removed from a group resulting, in some situations, in improved group performance. However, improvement is only predicted for cases in which the initial guesses of individuals in the group are biased.