The wisdom of many in few: Finding individuals who are as wise as the crowd

Is forecasting ability a stable trait? While domain knowledge and reasoning abilities are necessary for making accurate forecasts, research shows that knowing how accurate forecasters have been in the past is the best predictor of future accuracy. However, unlike the measurement of other traits, evaluating forecasting skill requires substantial time investment. Forecasters must make predictions about events that may not resolve for many days, weeks, months, or even years into the future before their accuracy can be estimated. Our work builds upon methods such as cultural consensus theory and proxy scoring rules to show talented forecasters can be discriminated in real time, without requiring any event resolutions. We define a peer similarity-based intersubjective evaluation method and test its utility in a unique longitudinal forecasting experiment. Because forecasters predicted all events at the same points in time, many of the confounds common to forecasting tournaments or observational data were eliminated. This allowed us to demonstrate the effectiveness of our method in real time, as time progressed and more information about forecasters became available. Intersubjective accuracy scores, which can be obtained immediately after the forecasts are made, were both valid and reliable estimators of forecasting talent. We also found that asking forecasters to make meta-predictions about what they expect others to believe can serve as an incentive-compatible method of intersubjective evaluation. Our results indicate that selecting small groups of, or even single forecasters, based on intersubjective accuracy can yield subsequent forecasts that approximate the actual accuracy of much larger crowd aggregates. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A survey of human judgement and quantitative forecasting methods

This paper's top-level goal is to provide an overview of research conducted in the many academic domains concerned with forecasting. By providing a summary encompassing these domains, this survey connects them, establishing a common ground for future discussions. To this end, we survey literature on human judgement and quantitative forecasting as well as hybrid methods that involve both humans and algorithmic approaches. The survey starts with key search terms that identified more than 280 publications in the fields of computer science, operations research, risk analysis, decision science, psychology and forecasting. Results show an almost 10-fold increase in the application-focused forecasting literature between the 1990s and the current decade, with a clear rise of quantitative, data-driven forecasting models. Comparative studies of quantitative methods and human judgement show that (1) neither method is universally superior, and (2) the better method varies as a function of factors such as availability, quality, extent and format of data, suggesting that (3) the two approaches can complement each other to yield more accurate and resilient models. We also identify four research thrusts in the human/machine-forecasting literature: (i) the choice of the appropriate quantitative model, (ii) the nature of the interaction between quantitative models and human judgement, (iii) the training and incentivization of human forecasters, and (iv) the combination of multiple forecasts (both algorithmic and human) into one. This review surveys current research in all four areas and argues that future research in the field of human/machine forecasting needs to consider all of them when investigating predictive performance. We also address some of the ethical dilemmas that might arise due to the combination of quantitative models with human judgement.

Too Much Trust in Group Decisions: Uncovering Hidden Profiles by Groups and Markets

A crucial challenge for organizations is to pool and aggregate information effectively. Traditionally, organizations have relied on committees and teams, but recently many organizations have explored the use of information markets. In this paper, the authors compared groups and markets in their ability to pool and aggregate information in a hidden-profiles task. In Study 1, groups outperformed markets when there were no conflicts of interest among participants, whereas markets outperformed groups when conflicts of interest were present. Also, participants had more trust in groups to uncover hidden profiles than in markets. Study 2 generalized these findings to a simple prediction task, confirming that people had more trust in groups than in markets. These results were not qualified by conflicts of interest. Drawing on experienced forecasters from Good Judgment Open, Study 3 found that familiarity and experience with markets increased the endorsement and use of markets relative to traditional committees.

Achieving More With Less: Intuitive Correction in Selection

Choosing between candidates for a position can be tricky, especially when the selection test is affected by irrelevant characteristics (e.g., reading speed). One can correct for this irrelevant attribute by penalizing individuals who have unjustifiably benefited from it. Statistical models do so by including the irrelevant attribute as a suppressor variable, but can people do the same without the help of a model? In three experiments (total N = 357), participants had to choose between two candidates, one of whom had higher levels of an irrelevant attribute and thus enjoyed an unfair advantage. Participants showed a substantial preference for the candidate with high levels of the irrelevant attribute, thus choosing the less suitable candidate. This bias was attenuated when the irrelevant attribute was a situational factor, probably by making the correction process more intuitive. Understanding the intuitive judgment of suppressor variables can help candidates from underprivileged groups boost their chances to succeed.

Does the perceived accuracy of urine drug testing impact clinical decision-making?

Background: Urine drug testing techniques have different rates of false-positive and false-negative test results. However, clinicians may have highly varying perceptions of test accuracy and may compensate for perceived inaccuracy by incorporating other factors into their interpretation of observed test results. Thus, there is the potential for adverse consequences from decisions based on inaccurate test results or interpretation. Methods: We surveyed 466 members of the American Society of Addiction Medicine to examine clinicians' perceptions of the accuracy of 2 types of urine drug tests, immunoassay (IA) and liquid chromatography-tandem mass spectrometry (LC-MS/MS), and the extent to which behavioral and demographic factors influence the interpretation of test results. Participants read 4 brief vignettes describing positive and negative test results in hypothetical patients who differed along several dimensions (gender, age, race/ethnicity, comorbid mental disorder, court-ordered versus voluntary status, treatment compliance). Outcome variables include likelihood of renewed drug use, likelihood of test error, whether to request additional testing, and whether to report the violation to a probation officer. Results: The strongest predictor of study outcomes was treatment compliance (consistent versus inconsistent attendance), as this was the only independent variable to generate effect sizes of medium strength. Significant effect sizes were also found for type of test used (IA versus LC-MS/MS), legal status (court-mandated versus voluntary), presence of a comorbid mental disorder, treatment history, and race, although effect sizes for these variables were small and less consistently observed. Conclusions: These results highlight the potential for error in clinician judgments about urine drug testing. Not only were participants likely to underestimate the accuracy of "confirmatory" LC-MS/MS testing, but vignettes suggested that a number of historical and demographic factors may influence interpretation of test results.

The Role of Type and Source of Uncertainty on the Processing of Climate Models Projections

Scientists agree that the climate is changing due to human activities, but there is less agreement about the specific consequences and their timeline. Disagreement among climate projections is attributable to the complexity of climate models that differ in their structure, parameters, initial conditions, etc. We examine how different sources of uncertainty affect people's interpretation of, and reaction to, information about climate change by presenting participants forecasts from multiple experts. Participants viewed three types of sets of sea-level rise projections: (1) precise, but conflicting; (2) imprecise, but agreeing, and (3) hybrid that were both conflicting and imprecise. They estimated the most likely sea-level rise, provided a range of possible values and rated the sets on several features - ambiguity, credibility, completeness, etc. In Study 1, everyone saw the same hybrid set. We found that participants were sensitive to uncertainty between sources, but not to uncertainty about which model was used. The impacts of conflict and imprecision were combined for estimation tasks and compromised for feature ratings. Estimates were closer to the experts' original projections, and sets were rated more favorably under imprecision. Estimates were least consistent with (narrower than) the experts in the hybrid condition, but participants rated the conflicting set least favorably. In Study 2, we investigated the hybrid case in more detail by creating several distinct interval sets that combine conflict and imprecision. Two factors drive perceptual differences: overlap - the structure of the forecast set (whether intersecting, nested, tangent, or disjoint) - and asymmetry - the balance of the set. Estimates were primarily driven by asymmetry, and preferences were primarily driven by overlap. Asymmetric sets were least consistent with the experts: estimated ranges were narrower, and estimates of the most likely value were shifted further below the set mean. Intersecting and nested sets were rated similarly to imprecision, and ratings of disjoint and tangent sets were rated like conflict. Our goal was to determine which underlying factors of information sets drive perceptions of uncertainty in consistent, predictable ways. The two studies lead us to conclude that perceptions of agreement require intersection and balance, and overly precise forecasts lead to greater perceptions of disagreement and a greater likelihood of the public discrediting and misinterpreting information.

Selecting an Equating Method: Linear or Equipercentile?

A new procedure for comparing results of linear and equipercentile equating methods is presented and illustrated. The proposed procedure requires (a) approximating the empirical score distributions of the two forms by means of the first terms of an infinite series, and (b) contrasting the results obtained when only the first two moments are used (i.e., linear equating) with more complete representations. The procedure is demonstrated by calculating the Kolmogorov-Smirnov and Cramer-Von Mises measures of discrepancy for a large number of unimodal distributions approximated by the Edgeworth and Gram-Charlier expansions. The results indicate that both statistics can be accurately predicted on the basis of the skewness and kurtosis of the two distributions. The most attractive feature of the proposed method is that it allows one to calculate the expected degree of efficiency in equating that can be achieved by linear equating, based on the moments of the two relevant distributions.

Variance Stabilizing Transformations and the Power of the F Test

The effect of the variance stabilizing transformations on the significance level and power of the F test, applied to Binomial and Poisson variables, was examined by a Monte Carlo study. The results indicate that the significance level is not affected by the transformations and that in most cases the power is not systematically increased by them,

Beyond Global Measures of Relative Importance: Some Insights from Dominance Analysis

The five articles in this special issue of Organizational Research Methods focus on various measures of global importance that rank and scale all p predictors in a multiple regression equation and compare their relative merits using a variety of real and simulated data sets. In this commentary, the authors describe several special features of dominance analysis that allow researchers to study more specific and narrowly focused aspects of relative importance. These analyses offer new insights into the pattern of importance that are not provided by any of the global measures of importance.

Analyzing Test-Taking Behavior: Decision Theory Meets Psychometric Theory

We investigate the implications of penalizing incorrect answers to multiple-choice tests, from the perspective of both test-takers and test-makers. To do so, we use a model that combines a well-known item response theory model with prospect theory (Kahneman and Tversky, Prospect theory: An analysis of decision under risk, Econometrica 47:263-91, 1979). Our results reveal that when test-takers are fully informed of the scoring rule, the use of any penalty has detrimental effects for both test-takers (they are always penalized in excess, particularly those who are risk averse and loss averse) and test-makers (the bias of the estimated scores, as well as the variance and skewness of their distribution, increase as a function of the severity of the penalty).

The Effect of the Raters' Marginal Distributions on Their Matched Agreement: A Rescaling Framework for Interpreting Kappa

Cohen's κ measures the improvement in classification above chance level and it is the most popular measure of interjudge agreement. Yet, there is considerable confusion about its interpretation. Specifically, researchers often ignore the fact that the observed level of matched agreement is bounded from above and below and the bounds are a function of the particular marginal distributions of the table. We propose that these bounds should be used to rescale the components of κ (observed and expected agreement). Rescaling κ in this manner results in κ', a measure that was originally proposed by Cohen (1960) and was largely ignored in both research and practice. This measure provides a common scale for agreement measures of tables with different marginal distributions. It reaches the maximal value of 1 when the judges show the highest level of agreement possible, given their marginal disagreements. We conclude that κ' should be used to measure the level of matched agreement contingent on a particular set of marginal distributions. The article provides a framework and a set of guidelines that facilitate comparisons between various types of agreement tables. We illustrate our points with simulations and real data from two studies-one involving judges' ratings of baseball players and one involving ratings of essays in high-stakes tests.

Verbal and numerical consumer recommendations: switching between recommendation formats leads to preference inconsistencies

Many Web sites provide consumers with product recommendations, which are typically presented by a sequence of verbal reviews and numerical ratings. In three experiments, we demonstrate that when participants switch between formats (e.g., from verbal to numerical), they are more prone to preference inconsistencies than when they aggregate the recommendations within the same format (e.g., verbal). When evaluating recommendations, participants rely primarily on central-location measures (e.g., mean) and less on other distribution characteristics (e.g., variance). We explain our findings within the theoretical framework of stimulus-response compatibility and we make practical recommendations for the design of recommendation systems and Web portals.

It’s a game of give and take: Modeling behavior in a give-or-take-some social dilemma

We investigate a new give-or-take-some (GOTS) dilemma paradigm that merges traditional give-some and take-some dilemmas. In this hybrid social dilemma, individuals can choose to give or to take resources from a shared resource pool. Previous empirical work by McCarter, Budescu, and Scheffran (2011) found that the composition of the group and the individuals’ endowments influenced their tendency to give and/or take. We reanalyze results from two experiments from McCarter, Budescu, et al. (2011) using the new paradigm and propose a simple model of individual behavior based on the players’ perceptions of their relative standing in the group and their perception of fair allocations. We also use these data to fit the model at the individual level and use it to provide a general framework for interpreting the group results.

Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change

The Intergovernmental Panel on Climate Change (IPCC) assesses information relevant to the understanding of climate change and explores options for adaptation and mitigation. The IPCC reports communicate uncertainty by using a set of probability terms accompanied by global interpretational guidelines. The judgment literature indicates that there are large differences in the way people understand such phrases, and that their use may lead to confusion and errors in communication. We conducted an experiment in which subjects read sentences from the 2007 IPCC report and assigned numerical values to the probability terms. The respondents' judgments deviated significantly from the IPCC guidelines, even when the respondents had access to these guidelines. These results suggest that the method used by the IPCC is likely to convey levels of imprecision that are too high. We propose an alternative form of communicating uncertainty, illustrate its effectiveness, and suggest several additional ways to improve the communication of uncertainty.

Improving communication of uncertainty in the reports of the intergovernmental panel on climate change

The Intergovernmental Panel on Climate Change (IPCC) assesses information relevant to the understanding of climate change and explores options for adaptation and mitigation. The IPCC reports communicate uncertainty by using a set of probability terms accompanied by global interpretational guidelines. The judgment literature indicates that there are large differences in the way people understand such phrases, and that their use may lead to confusion and errors in communication. We conducted an experiment in which subjects read sentences from the 2007 IPCC report and assigned numerical values to the probability terms. The respondents' judgments deviated significantly from the IPCC guidelines, even when the respondents had access to these guidelines. These results suggest that the method used by the IPCC is likely to convey levels of imprecision that are too high. We propose an alternative form of communicating uncertainty, illustrate its effectiveness, and suggest several additional ways to improve the communication of uncertainty.

Collective induction without cooperation? Learning and knowledge transfer in cooperative groups and competitive auctions

There is strong evidence that groups perform better than individuals do on intellective tasks with demonstrably correct solutions. Typically, these studies assume that group members share common goals. The authors extend this line of research by replacing standard face-to-face group interactions with competitive auctions, allowing for conflicting individual incentives. In a series of studies involving the well-known Wason selection task, they demonstrate that competitive auctions induce learning effects equally impressive as those of standard group interactions, and they uncover specific and general knowledge transfers from these institutions to new reasoning problems. The authors identify payoff feedback and information pooling as the driving factors underlying these findings, and they explain these factors within the theoretical framework of collective induction.

Evidence that androgenic and estrogenic metabolites contribute to the effects of dehydroepiandrosterone on cognition in postmenopausal women

Prior studies of the effects of dehydroepiandrosterone (DHEA) on cognition have produced complex and inconsistent results. We hypothesize that these results may arise, in part, because of DHEA's metabolism into estrogens and androgens that produce opposing effects on cognition. Our study administered 50 mg of oral DHEA daily for 4 weeks in a placebo-controlled crossover design to six postmenopausal women. We measured blood levels of androgens (total testosterone, free testosterone, DHEA, DHEAS), estrogens (estradiol, estrone), and cognitive performance on recognition memory, perceptual identification, digit span memory, and visual attentional vigilance under both drug and placebo conditions. Multiple regression models incorporating the factors of age and body mass index (BMI) were used to ascertain the relation between sex steroids and cognitive performance. Our results demonstrated that estrogens produced a positive effect on recognition memory, while androgens produced a negative effect. This pattern reversed in perceptual identification with estrogens producing a negative effect and androgens producing a positive effect. In addition, BMI produced a negative effect on digit span memory, age produced a negative effect on perceptual identification, and androgens produced a negative effect on visual attentional vigilance. These results help, in part, to explain DHEA's complex effects on cognition. The diverse effects of sex steroids across tasks underscore the importance of identifying the specific cognitive mechanisms influenced by sex steroids and emphasizes that one should not expect sex steroids to produce homogeneous effects across cognitive tasks.

You Say "Probable" and I Say "Likely": Improving Interpersonal Communication With Verbal Probability Phrases

When forecasters and decision makers describe uncertain events using verbal probability terms, there is a risk of miscommunication because people use different probability phrases and interpret them in different ways. In an effort to facilitate the communication process, the authors investigated various ways of converting the forecasters' verbal probabilities to the decision maker's terms. The authors present 3 studies in which participants judged the probabilities of distinct events using both numerical and verbal probabilities. They define several indexes of interindividual coassignment of phrases to the same events and evaluate the conversion methods by comparing the values of these indexes for the converted and the unconverted judgments. In all the cases studied, the conversion methods significantly reduced the error rates in communicating uncertainties.