Blinded with Science or Informed by Charts? A Replication Study

How Difference Tasks Are Affected by Probability Format, Part 2: A Making Numbers Meaningful Systematic Review

Background. To evaluate the effect of data presentation format on communication of health probabilities, the Making Numbers Meaningful team undertook a systematic review. Purpose. This article presents evidence about difference tasks, in which a reader examines information to evaluate differences between probabilities, such as the effect of a therapy on the chance of recurrence. This article covers the effect of format on 5 outcomes: 1) perceptions of or feelings about effectiveness, 2) behavioral intentions or behaviors, 3) trust, 4) preference for the format, and 5) discrimination. Data Sources. MEDLINE, Embase, CINAHL, the Cochrane Library, PsycINFO, ERIC, ACM Digital Library; hand search. Finding Selection. Experimental/quasi-experimental studies comparing 2 or more formats for presenting quantitative health information. This article covers 205 findings from 101 unique studies reported in 84 articles. Data Extraction. Dual extraction of information on stimulus, task, and perceptual, affective, cognitive, and behavioral outcomes. Data Synthesis. Evidence is moderate to strong that behavioral intention is affected more by relative differences than absolute ones, by numerator-only graphics than part-to-whole graphics, by messages with anecdotes than without, and by information about what others chose. Evidence is strong that perceived and felt effectiveness is affected more by relative differences than by absolute ones and more by numerator-only graphics rather than part-to-whole graphics. For graphic preferences, bar charts were preferred to icon arrays and graphics with data labels to graphics without. Other comparisons had weak or insufficient evidence. Limitations. The detailed approach to evidence syntheses provides narrowly targeted evidence rather than broad statements. Conclusions. Moderate to strong evidence can be derived on effects of probability difference format on behavioral intention, perceived or felt effectiveness, and preference for format.

Highlights

Communicating relative risk differences as opposed to absolute risk differences, using numerator-only instead of part-to-whole graphics, and including anecdotes or information about others' decisions will all increase intentions to engage in a behavior.Relative risks (rather than absolute risk differences) and numerator-only graphics (rather than part-to-whole) will also increase felt and perceived effectiveness.To illustrate probability differences, people tend to prefer bar charts over icon arrays and graphics with labels over those without.All findings regarding the impact of different presentation formats for probability differences on trust produced insufficient evidence.

User Experience of Visualizations in Motion: A Case Study and Design Considerations

We present a systematic review, an empirical study, and a first set of considerations for designing visualizations in motion, derived from a concrete scenario in which these visualizations were used to support a primary task. In practice, when viewers are confronted with embedded visualizations, they often have to focus on a primary task and can only quickly glance at a visualization showing rich, often dynamically updated, information. As such, the visualizations must be designed so as not to distract from the primary task, while at the same time being readable and useful for aiding the primary task. For example, in games, players who are engaged in a battle have to look at their enemies but also read the remaining health of their own game character from the health bar over their character's head. Many trade-ofts are possible in the design of embedded visualizations in such dynamic scenarios, which we explore in-depth in this paper with a focus on user experience. We use video games as an example of an application context with a rich existing set of visualizations in motion. We begin our work with a systematic review of in-game visualizations in motion. Next, we conduct an empirical user study to investigate how different embedded visualizations in motion designs impact user experience. We conclude with a set of considerations and trade-offs for designing visualizations in motion more broadly as derived from what we learned about video games. All supplemental materials of this paper are available at osf.io/3v8wm/.

Scope, Methods, and Overview Findings for the Making Numbers Meaningful Evidence Review of Communicating Probabilities in Health: A Systematic Review

Background. The format in which probabilities are presented influences comprehension and interpretation. Purpose. To develop comprehensive evidence-based guidance about how to communicate probabilities in health and to identify strengths and weaknesses in the literature. This article presents methods for the review of probability communication and is accompanied by several results articles. Data Sources. MEDLINE, Embase, CINAHL, the Cochrane Library, PsycINFO, ERIC, ACM Digital Library; hand search of 4 journals. Study Selection. Two reviewers conducted screening to identify experimental and quasi-experimental research that compared 2 or more formats for presenting quantitative health information to patients or lay audiences. Data Extraction. In our conceptual framework, people make sense of a stimulus (data in a data presentation format) by performing cognitive tasks, resulting in perceptual, affective, cognitive, or behavioral responses measured as 1 of 14 distinct outcomes. The study team developed custom instruments to extract concepts, conduct risk-of-bias evaluation, and evaluate individual findings for credibility. Data Synthesis. Findings were grouped into tables by task and outcome for evidence synthesis. Limitations. Reviewer error could have led to missing relevant studies despite having 2 independent reviewers screening each article. The granular data extraction and syntheses slowed the work and may have made it less replicable. Credibility was evaluated by only 2 experts. Conclusions. After reviewing 26,793 titles and abstracts, we identified 316 articles about probability communication. Data extraction produced 1,119 individual findings, which were grouped into 37 evidence tables, each containing evidence on up to 10 data presentation format comparisons. The Making Numbers Meaningful project required novel methods for classifying and synthesizing research, which reveal patterns of strength and weakness in the probability communication literature.

Highlights

The Making Numbers Meaningful project conducted a comprehensive systematic review of experimental and quasi-experimental research that compared 2 or more formats for presenting quantitative health information to patients or other lay audiences. The current article focuses on probability information.Based on a conceptual taxonomy, we reviewed studies based on the cognitive tasks required of participants, assessing 14 distinct possible outcomes.Our review identified 316 articles involving probability communications that generated 1,119 distinct research findings, each of which was reviewed by multiple experts for credibility.The overall pattern of findings highlights which probability communication questions have been well researched and which have not. For example, there has been far more research on communicating single probabilities than on communicating more complex information such as trends over time, and there has been a large amount of research on the effect of communication approaches on behavioral intentions but relatively little on behaviors.

How Point (Single-Probability) Tasks Are Affected by Probability Format, Part 1: A Making Numbers Meaningful Systematic Review

Background. To create guidance on the effect of data presentation format on communication of health numbers, the Making Numbers Meaningful project undertook a systematic review. Purpose. This article (one of a series) covers research studying so-called "point tasks," in which a reader examines stimuli to obtain information about single probabilities. The current article presents the evidence on the effects of data presentation format on multiple outcomes: identification and recall, contrast, categorization, and computation. Data Sources. MEDLINE, Embase, CINAHL, the Cochrane Library, PsycINFO, ERIC, ACM Digital Library; hand search of 4 journals. Finding Selection. Manual pairwise screening to identify experimental and quasi-experimental research comparing 2 or more formats for quantitative health information for patients or other lay audiences. This article reports on 218 findings from 99 articles on single probability communication. Data Extraction. Pairwise extraction of data on stimulus (data in a data presentation format), task, and perceptual/affective/cognitive/behavioral outcomes. Data Synthesis. Most evidence on these outcomes was weak or insufficient. There was moderate to strong evidence that 1) recall was better with icon arrays with human figures than icon arrays with blocks, 2) survival curves make it easier to identify points of highest survival than mortality curves (contrast outcome), 3) adding an average population probability to a message about an individual probability may not affect recall, 4) computation performance is better with bar charts combined with data labels than with either numbers or graphics alone, 5) computation performance with rates is better when denominators match, and 6) framing strongly affects risky choices (contrast). Limitations. Heterogeneous study designs reduced the ability to develop strong evidence. Conclusions. Few findings assessing identification or recall, contrast, categorization, or computation outcomes for point tasks were comparable enough to each other to generate strong evidence.

Highlights

Many researchers have studied the effects of data presentation formats of single probabilities on different outcomes.However, few findings are comparable enough to allow for strong evidence-based conclusions about the impact on identification, recall, contrast, categorization, and computation outcomes.

Icon arrays reduce concern over COVID-19 vaccine side effects: a randomized control study

On April 13, 2021, the CDC announced that the administration of Johnson and Johnson's COVID-19 vaccine would be paused due to a rare blood clotting side effect in ~ 0.0001% of people given the vaccine. Most people who are hesitant to get a COVID-19 vaccine list potential side effects as their main concern (PEW, 2021); thus, it is likely that this announcement increased vaccine hesitancy among the American public. Two days after the CDC's announcement, we administered a survey to a group of 2,046 Americans to assess their changes in attitudes toward COVID-19 vaccines. The aim of this study was to investigate whether viewing icon arrays of side effect risk would prevent increases in COVID-19 vaccine hesitancy due to the announcement. We found that using icon arrays to illustrate the small chance of experiencing the blood clotting side effect significantly prevented increases in aversion toward the Johnson and Johnson vaccine as well as all other COVID-19 vaccines.

Revisited: Comparison of Empirical Methods to Evaluate Visualizations Supporting Crafting and Assembly Purposes

Ubiquitous, situated, and physical visualizations create entirely new possibilities for tasks contextualized in the real world, such as doctors inserting needles. During the development of situated visualizations, evaluating visualizations is a core requirement. However, performing such evaluations is intrinsically hard as the real scenarios are safety-critical or expensive to test. To overcome these issues, researchers and practitioners adapt classical approaches from ubiquitous computing and use surrogate empirical methods such as Augmented Reality (AR), Virtual Reality (VR) prototypes, or merely online demonstrations. This approach's primary assumption is that meaningful insights can also be gained from different, usually cheaper and less cumbersome empirical methods. Nevertheless, recent efforts in the Human-Computer Interaction (HCI) community have found evidence against this assumption, which would impede the use of surrogate empirical methods. Currently, these insights rely on a single investigation of four interactive objects. The goal of this work is to investigate if these prior findings also hold for situated visualizations. Therefore, we first created a scenario where situated visualizations support users in do-it-yourself (DIY) tasks such as crafting and assembly. We then set up five empirical study methods to evaluate the four tasks using an online survey, as well as VR, AR, laboratory, and in-situ studies. Using this study design, we conducted a new study with 60 participants. Our results show that the situated visualizations we investigated in this study are not prone to the same dependency on the empirical method, as found in previous work. Our study provides the first evidence that analyzing situated visualizations through different empirical (surrogate) methods might lead to comparable results.