Conceptual and Methodological Issues in Evaluating Multidimensional Visualizations for Decision Support

Decoupling Judgment and Decision Making: A Tale of Two Tails

Is it true that if citizens understand hurricane probabilities, they will make more rational decisions for evacuation? Finding answers to such questions is not straightforward in the literature because the terms "judgment" and "decision making" are often used interchangeably. This terminology conflation leads to a lack of clarity on whether people make suboptimal decisions because of inaccurate judgments of information conveyed in visualizations or because they use alternative yet currently unknown heuristics. To decouple judgment from decision making, we review relevant concepts from the literature and present two preregistered experiments (N = 601) to investigate if the task (judgment versus decision making), the scenario (sports versus humanitarian), and the visualization (quantile dotplots, density plots, probability bars) affect accuracy. While experiment 1 was inconclusive, we found evidence for a difference in experiment 2. Contrary to our expectations and previous research, which found decisions less accurate than their direct-equivalent judgments, our results pointed in the opposite direction. Our findings further revealed that decisions were less vulnerable to status-quo bias, suggesting decision makers may disfavor responses associated with inaction. We also found that both scenario and visualization types can influence people's judgments and decisions. Although effect sizes are not large and results should be interpreted carefully, we conclude that judgments cannot be safely used as proxy tasks for decision making, and discuss implications for visualization research and beyond. Materials and preregistrations are available at https://osf.io/ufzp5/?view_only=adc0f78a23804c31bf7fdd9385cb264f.

From Information to Choice: A Critical Inquiry Into Visualization Tools for Decision Making

In the face of complex decisions, people often engage in a three-stage process that spans from (1) exploring and analyzing pertinent information (intelligence); (2) generating and exploring alternative options (design); and ultimately culminating in (3) selecting the optimal decision by evaluating discerning criteria (choice). We can fairly assume that all good visualizations aid in the "intelligence" stage by enabling data exploration and analysis. Yet, to what degree and how do visualization systems currently support the other decision making stages, namely "design" and "choice"? To further explore this question, we conducted a comprehensive review of decision-focused visualization tools by examining publications in major visualization journals and conferences, including VIS, EuroVis, and CHI, spanning all available years. We employed a deductive coding method and in-depth analysis to assess whether and how visualization tools support design and choice. Specifically, we examined each visualization tool by (i) its degree of visibility for displaying decision alternatives, criteria, and preferences, and (ii) its degree of flexibility for offering means to manipulate the decision alternatives, criteria, and preferences with interactions such as adding, modifying, changing mapping, and filtering. Our review highlights the opportunities and challenges that decision-focused visualization tools face in realizing their full potential to support all stages of the decision making process. It reveals a surprising scarcity of tools that support all stages, and while most tools excel in offering visibility for decision criteria and alternatives, the degree of flexibility to manipulate these elements is often limited, and the lack of tools that accommodate decision preferences and their elicitation is notable. Based on our findings, to better support the choice stage, future research could explore enhancing flexibility levels and variety, exploring novel visualization paradigms, increasing algorithmic support, and ensuring that this automation is user-controlled via the enhanced flexibility I evels. Our curated list of the 88 surveyed visualization tools is available in the OSF link (https://osf.io/nrasz/?view_only=b92a90a34ae241449b5f2cd33383bfcb).

COMPO*SED: Composite Parallel Coordinates for Co-Dependent Multi-Attribute Choices

We propose Composite Parallel Coordinates, a novel parallel coordinates technique to effectively represent the interplay of component alternatives in a system. It builds upon a dedicated data model that formally describes the interaction of components. Parallel coordinates can help decision-makers identify the most preferred solution among a number of alternatives. Multi-component systems require one such multi-attribute choice for each component. Each of these choices might have side effects on the system's operability and performance, making them co-dependent. Common approaches employ complex multi-component models or involve back-and-forth iterations between single components until an acceptable compromise is reached. A simultaneous visual exploration across independently modeled but connected components is needed to make system design more efficient. Using dedicated layout and interaction strategies, our Composite Parallel Coordinates allow analysts to explore both individual properties of components as well as their interoperability and joint performance. We showcase the effectiveness of Composite Parallel Coordinates for co-dependent multi-attribute choices by means of three real-world scenarios from distinct application areas. In addition to the case studies, we reflect on observing two domain experts collaboratively working with the proposed technique and communicating along the way.

A Survey of Perception-Based Visualization Studies by Task

Knowledge of human perception has long been incorporated into visualizations to enhance their quality and effectiveness. The last decade, in particular, has shown an increase in perception-based visualization research studies. With all of this recent progress, the visualization community lacks a comprehensive guide to contextualize their results. In this report, we provide a systematic and comprehensive review of research studies on perception related to visualization. This survey reviews perception-focused visualization studies since 1980 and summarizes their research developments focusing on low-level tasks, further breaking techniques down by visual encoding and visualization type. In particular, we focus on how perception is used to evaluate the effectiveness of visualizations, to help readers understand and apply the principles of perception of their visualization designs through a task-optimized approach. We concluded our report with a summary of the weaknesses and open research questions in the area.

The Unmet Data Visualization Needs of Decision Makers Within Organizations

When an organization chooses one course of action over alternatives, this task typically falls on a decision maker with relevant knowledge, experience, and understanding of context. Decision makers rely on data analysis, which is either delegated to analysts, or done on their own. Often the decision maker combines data, likely uncertain or incomplete, with non-formalized knowledge within a multi-objective problem space, weighing the recommendations of analysts within broader contexts and goals. As most past research in visual analytics has focused on understanding the needs and challenges of data analysts, less is known about the tasks and challenges of organizational decision makers, and how visualization support tools might help. Here we characterize the decision maker as a domain expert, review relevant literature in management theories, and report the results of an empirical survey and interviews with people who make organizational decisions. We identify challenges and opportunities for novel visualization tools, including trade-off overviews, scenario-based analysis, interrogation tools, flexible data input and collaboration support. Our findings stress the need to expand visualization design beyond data analysis into tools for information management.

Improving Visualization Design for Effective Multi-Objective Decision Making

Decision-makers across many professions are often required to make multi-objective decisions over increasingly larger volumes of data with several competing criteria. Data visualization is a powerful tool for exploring these complex 'solution spaces', but there is limited research on its ability to support multi-objective decisions. In this article, we explore the effects of chart complexity and data volume on decision quality in multi-objective scenarios with complex trade-offs. We look at the impact of four common multidimensional chart types (scatter plot matrices, parallel coordinates plots, heat maps, radar charts), the number of options and dimensions and participant chart usage experience on decision time and accuracy when selecting the 'optimal option'. As objectively evaluating the quality of multi-objective decisions and the trade-offs involved is challenging, we employ rank- and score-based accuracy metrics. While heat maps demonstrate a time advantage, our findings show no strong performance benefit for one chart type over another for accuracy. We find mixed evidence for the impact of chart complexity on performance, with our results suggesting the existence of a 'ceiling' in the number of dimensions considered by participants. This points to a potential limit to data complexity that is useful for decision making. Lastly, participants who use charts frequently performed better, suggesting that users can potentially be trained to effectively use complex visualizations in their decision-making.

A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?

It has been widely suggested that a key goal of visualization systems is to assist decision making, but is this true? We conduct a critical investigation on whether the activity of decision making is indeed central to the visualization domain. By approaching decision making as a user task, we explore the degree to which decision tasks are evident in visualization research and user studies. Our analysis suggests that decision tasks are not commonly found in current visualization task taxonomies and that the visualization field has yet to leverage guidance from decision theory domains on how to study such tasks. We further found that the majority of visualizations addressing decision making were not evaluated based on their ability to assist decision tasks. Finally, to help expand the impact of visual analytics in organizational as well as casual decision making activities, we initiate a research agenda on how decision making assistance could be elevated throughout visualization research.

Impacts of Visualizations on Decoy Effects

The decoy effect is a well-known, intriguing decision-making bias that is often exploited by marketing practitioners to steer consumers towards a desired purchase outcome. It demonstrates that an inclusion of an alternative in the choice set can alter one’s preference among the other choices. Although this decoy effect has been universally observed in the real world and also studied by many economists and psychologists, little is known about how to mitigate the decoy effect and help consumers make informed decisions. In this study, we conducted two experiments: a quantitative experiment with crowdsourcing and a qualitative interview study—first, the crowdsourcing experiment to see if visual interfaces can help alleviate this cognitive bias. Four types of visualizations, one-sided bar chart, two-sided bar charts, scatterplots, and parallel-coordinate plots, were evaluated with four different types of scenarios. The results demonstrated that the two types of bar charts were effective in decreasing the decoy effect. Second, we conducted a semi-structured interview to gain a deeper understanding of the decision-making strategies while making a choice. We believe that the results have an implication on showing how visualizations can have an impact on the decision-making process in our everyday life.

A Visual Analytics System for Oil Spill Response and Recovery

Extensive research has been done on oil spill simulation techniques, spatial optimization models, and oil spill cleanup strategies. This article presents a visual analytics system that integrates the independent facets of spill modeling techniques and spatial optimization to enable inspection, exploration, and decision making for offshore oil spill response.

AgentVis: Visual Analysis of Agent Behavior With Hierarchical Glyphs

Glyphs representing complex behavior provide a useful and common means of visualizing multivariate data. However, due to their complex shape, overlapping, and occlusion of glyphs is a common and prominent limitation. This limits the number of discreet data tuples that can be displayed in a given image. Using a real-world application, glyphs are used to depict agent behavior in a call center. However, many call centers feature thousands of agents. A standard approach representing thousands of agents with glyphs does not scale. To accommodate the visualization incorporating thousands of glyphs we develop clustering of overlapping glyphs into a single parent glyph. This hierarchical glyph represents the mean value of all child agent glyphs, removing overlap and reduTcing visual clutter. Multi-variate clustering techniques are explored and developed in collaboration with domain experts in the call center industry. We implement dynamic control of glyph clusters according to zoom level and customized distance metrics, to utilize image space with reduced overplotting and cluttering. We demonstrate our technique with examples and a usage scenario using real-world call-center data to visualize thousands of call center agents, revealing insight into their behavior and reporting feedback from expert call-center analysts.

Does visual saliency affect decision-making?

Abstract

In the present study, we explore potential effects of visual saliency on decision quality in context of multi-criteria decision-making (MCDM). We compare two visualization techniques: parallel coordinates (PC) and scatterplot matrices (SPM). We investigate the impact of saliency facilitated by means of either color or size. The saliency and visualization techniques were factors in our analysis, and effects were evaluated in terms of decision quality, attention, time on task, and confidence. Results show that the quality of choice and attention were comparable for all saliency conditions when SPM was used. For PC, we found a positive effect of color saliency both on the quality of choice and on attention. Different forms of saliency led to varying times on task in both PC and SPM; however, those variations were not significant. A comparison of PC and SPM shows, users spent less time on the task, obtained better decision quality, and were more confident with their decision when using PC. To summarize, our findings suggest that saliency can increase attention and decision quality in MCDM for certain visualization techniques and forms of saliency. Another contribution of this work is the novel suggestion of the method to elicit of users’ preferences; its potential benefits are discussed in the end of the paper.

Graphic abstract

A Task-Based Taxonomy of Cognitive Biases for Information Visualization

Information visualization designers strive to design data displays that allow for efficient exploration, analysis, and communication of patterns in data, leading to informed decisions. Unfortunately, human judgment and decision making are imperfect and often plagued by cognitive biases. There is limited empirical research documenting how these biases affect visual data analysis activities. Existing taxonomies are organized by cognitive theories that are hard to associate with visualization tasks. Based on a survey of the literature we propose a task-based taxonomy of 154 cognitive biases organized in 7 main categories. We hope the taxonomy will help visualization researchers relate their design to the corresponding possible biases, and lead to new research that detects and addresses biased judgment and decision making in data visualization.

Task-Based Effectiveness of Basic Visualizations

Visualizations of tabular data are widely used; understanding their effectiveness in different task and data contexts is fundamental to scaling their impact. However, little is known about how basic tabular data visualizations perform across varying data analysis tasks. In this paper, we report results from a crowdsourced experiment to evaluate the effectiveness of five small scale (5-34 data points) two-dimensional visualization types-Table, Line Chart, Bar Chart, Scatterplot, and Pie Chart-across ten common data analysis tasks using two datasets. We find the effectiveness of these visualization types significantly varies across task, suggesting that visualization design would benefit from considering context-dependent effectiveness. Based on our findings, we derive recommendations on which visualizations to choose based on different tasks. We finally train a decision tree on the data we collected to drive a recommender, showcasing how to effectively engineer experimental user data into practical visualization systems.

A Study on Visual Representations for Active Plant Wall Data Analysis

The indoor climate is closely related to human health, well-being, and comfort. Thus, an understanding of the indoor climate is vital. One way to improve the indoor climates is to place an aesthetically pleasing active plant wall in the environment. By collecting data using sensors placed in and around the plant wall both the indoor climate and the status of the plant wall can be monitored and analyzed. This manuscript presents a user study with domain experts in this field with a focus on the representation of such data. The experts explored this data with a Line graph, a Horizon graph, and a Stacked area graph to better understand the status of the active plant wall and the indoor climate. Qualitative measures were collected with Think-aloud protocol and semi-structured interviews. The study resulted in four categories of analysis tasks: Overview, Detail, Perception, and Complexity. The Line graph was found to be preferred for use in providing an overview, and the Horizon graph for detailed analysis, revealing patterns and showing discernible trends, while the Stacked area graph was generally not preferred. Based on these findings, directions for future research are discussed and formulated. The results and future directions of this research can facilitate the analysis of multivariate temporal data, both for domain users and visualization researchers.

Mitigating the Attraction Effect with Visualizations

Human decisions are prone to biases, and this is no less true for decisions made within data visualizations. Bias mitigation strategies often focus on the person, by educating people about their biases, typically with little success. We focus instead on the system, presenting the first evidence that altering the design of an interactive visualization tool can mitigate a strong bias - the attraction effect. Participants viewed 2D scatterplots where choices between superior alternatives were affected by the placement of other suboptimal points. We found that highlighting the superior alternatives weakened the bias, but did not eliminate it. We then tested an interactive approach where participants completely removed locally dominated points from the view, inspired by the elimination by aspects strategy in the decision-making literature. This approach strongly decreased the bias, leading to a counterintuitive suggestion: tools that allow removing inappropriately salient or distracting data from a view may help lead users to make more rational decisions.

Evaluating Multi-Dimensional Visualizations for Understanding Fuzzy Clusters

Fuzzy clustering assigns a probability of membership for a datum to a cluster, which veritably reflects real-world clustering scenarios but significantly increases the complexity of understanding fuzzy clusters. Many studies have demonstrated that visualization techniques for multi-dimensional data are beneficial to understand fuzzy clusters. However, no empirical evidence exists on the effectiveness and efficiency of these visualization techniques in solving analytical tasks featured by fuzzy clusters. In this paper, we conduct a controlled experiment to evaluate the ability of fuzzy clusters analysis to use four multi-dimensional visualization techniques, namely, parallel coordinate plot, scatterplot matrix, principal component analysis, and Radviz. First, we define the analytical tasks and their representative questions specific to fuzzy clusters analysis. Then, we design objective questionnaires to compare the accuracy, time, and satisfaction in using the four techniques to solve the questions. We also design subjective questionnaires to collect the experience of the volunteers with the four techniques in terms of ease of use, informativeness, and helpfulness. With a complete experiment process and a detailed result analysis, we test against four hypotheses that are formulated on the basis of our experience, and provide instructive guidance for analysts in selecting appropriate and efficient visualization techniques to analyze fuzzy clusters.