On measuring agreement with numerically bounded linguistic probability schemes: A re-analysis of data from Wintle, Fraser, Wills, Nicholson, and Fidler (2019)

Communicating uncertainty in national security intelligence: Expert and nonexpert interpretations of and preferences for verbal and numeric formats

Organizations in several domains including national security intelligence communicate judgments under uncertainty using verbal probabilities (e.g., likely) instead of numeric probabilities (e.g., 75% chance), despite research indicating that the former have variable meanings across individuals. In the intelligence domain, uncertainty is also communicated using terms such as low, moderate, or high to describe the analyst's confidence level. However, little research has examined how intelligence professionals interpret these terms and whether they prefer them to numeric uncertainty quantifiers. In two experiments (N = 481 and 624, respectively), uncertainty communication preferences of expert (n = 41 intelligence analysts in Experiment 1) and nonexpert intelligence consumers were elicited. We examined which format participants judged to be more informative and simpler to process. We further tested whether participants treated verbal probability and confidence terms as independent constructs and whether participants provided coherent numeric probability translations of verbal probabilities. Results showed that although most nonexperts favored the numeric format, experts were about equally split, and most participants in both samples regarded the numeric format as more informative. Experts and nonexperts consistently conflated probability and confidence. For instance, confidence intervals inferred from verbal confidence terms had a greater effect on the location of the estimate than the width of the estimate, contrary to normative expectation. Approximately one-fourth of experts and over one-half of nonexperts provided incoherent numeric probability translations for the terms likely and unlikely when the elicitation of best estimates and lower and upper bounds were briefly spaced by intervening tasks.

Facilitating sender-receiver agreement in communicated probabilities: Is it best to use words, numbers or both?

Organizations tasked with communicating expert judgments couched in uncertainty often use numerically bounded linguistic probability schemes to standardize the meaning of verbal probabilities. An experiment (N = 1,202) was conducted to ascertain whether agreement with such a scheme was better when probabilities were presented verbally, numerically or in a combined “verbal + numeric” format. Across three agreement measures, the numeric and combined formats outperformed the verbal format and also yielded better discrimination between low and high probabilities and were less susceptible to the fifty-fifty blip phenomenon. The combined format did not confer any advantage over the purely numeric format. The findings indicate that numerically bounded linguistic probability schemes are an ineffective means of communicating information about probabilities to others and they call into question recommendations for use of the combined format for delivering such schemes.