ViDX: Visual Diagnostics of Assembly Line Performance in Smart Factories

IVESA - Visual Analysis of Time-Stamped Event Sequences

Time-stamped event sequences (TSEQs) are time-oriented data without value information, shifting the focus of users to the exploration of temporal event occurrences. TSEQs exist in application domains, such as sleeping behavior, earthquake aftershocks, and stock market crashes. Domain experts face four challenges, for which they could use interactive and visual data analysis methods. First, TSEQs can be large with respect to both the number of sequences and events, often leading to millions of events. Second, domain experts need validated metrics and features to identify interesting patterns. Third, after identifying interesting patterns, domain experts contextualize the patterns to foster sensemaking. Finally, domain experts seek to reduce data complexity by data simplification and machine learning support. We present IVESA, a visual analytics approach for TSEQs. It supports the analysis of TSEQs at the granularities of sequences and events, supported with metrics and feature analysis tools. IVESA has multiple linked views that support overview, sort+filter, comparison, details-on-demand, and metadata relation-seeking tasks, as well as data simplification through feature analysis, interactive clustering, filtering, and motif detection and simplification. We evaluated IVESA with three case studies and a user study with six domain experts working with six different datasets and applications. Results demonstrate the usability and generalizability of IVESA across applications and cases that had up to 1,000,000 events.

HiRegEx: Interactive Visual Query and Exploration of Multivariate Hierarchical Data

When using exploratory visual analysis to examine multivariate hierarchical data, users often need to query data to narrow down the scope of analysis. However, formulating effective query expressions remains a challenge for multivariate hierarchical data, particularly when datasets become very large. To address this issue, we develop a declarative grammar, HiRegEx (Hierarchical data Regular Expression), for querying and exploring multivariate hierarchical data. Rooted in the extended multi-level task topology framework for tree visualizations (e-MLTT), HiRegEx delineates three query targets (node, path, and subtree) and two aspects for querying these targets (features and positions), and uses operators developed based on classical regular expressions for query construction. Based on the HiRegEx grammar, we develop an exploratory framework for querying and exploring multivariate hierarchical data and integrate it into the TreeQueryER prototype system. The exploratory framework includes three major components: top-down pattern specification, bottom-up data-driven inquiry, and context-creation data overview. We validate the expressiveness of HiRegEx with the tasks from the e-MLTT framework and showcase the utility and effectiveness of TreeQueryER system through a case study involving expert users in the analysis of a citation tree dataset.

Internet of Things Data Visualization for Business Intelligence

This study contributes to the research on Internet of Things data visualization for business intelligence processes, an area of growing interest to scholars, by conducting a systematic review of the literature. A total of 237 articles published over the past 11 years were obtained and compared. This made it possible to identify the top contributing and most influential authors, countries, publishers, institutions, papers, and research findings, together with the challenges facing current research. Based on these results, this work provides a thorough insight into the field by proposing four research categories (Technology infrastructure, Case examples, Final-user experience, and Big Data tools), together with the development of these research streams over time and their future research directions.

ECoalVis: Visual Analysis of Control Strategies in Coal-fired Power Plants

Improving the efficiency of coal-fired power plants has numerous benefits. The control strategy is one of the major factors affecting such efficiency. However, due to the complex and dynamic environment inside the power plants, it is hard to extract and evaluate control strategies and their cascading impact across massive sensors. Existing manual and data-driven approaches cannot well support the analysis of control strategies because these approaches are time-consuming and do not scale with the complexity of the power plant systems. Three challenges were identified: a) interactive extraction of control strategies from large-scale dynamic sensor data, b) intuitive visual representation of cascading impact among the sensors in a complex power plant system, and c) time-lag-aware analysis of the impact of control strategies on electricity generation efficiency. By collaborating with energy domain experts, we addressed these challenges with ECoalVis, a novel interactive system for experts to visually analyze the control strategies of coal-fired power plants extracted from historical sensor data. The effectiveness of the proposed system is evaluated with two usage scenarios on a real-world historical dataset and received positive feedback from experts.

Survey on Visual Analysis of Event Sequence Data

Event sequence data record series of discrete events in the time order of occurrence. They are commonly observed in a variety of applications ranging from electronic health records to network logs, with the characteristics of large-scale, high-dimensional and heterogeneous. This high complexity of event sequence data makes it difficult for analysts to manually explore and find patterns, resulting in ever-increasing needs for computational and perceptual aids from visual analytics techniques to extract and communicate insights from event sequence datasets. In this paper, we review the state-of-the-art visual analytics approaches, characterize them with our proposed design space, and categorize them based on analytical tasks and applications. From our review of relevant literature, we have also identified several remaining research challenges and future research opportunities.

Visual Drift Detection for Event Sequence Data of Business Processes

Event sequence data is increasingly available in various application domains, such as business process management, software engineering, or medical pathways. Processes in these domains are typically represented as process diagrams or flow charts. So far, various techniques have been developed for automatically generating such diagrams from event sequence data. An open challenge is the visual analysis of drift phenomena when processes change over time. In this article, we address this research gap. Our contribution is a system for fine-granular process drift detection and corresponding visualizations for event logs of executed business processes. We evaluated our system both on synthetic and real-world data. On synthetic logs, we achieved an average F-score of 0.96 and outperformed all the state-of-the-art methods. On real-world logs, we identified all types of process drifts in a comprehensive manner. Finally, we conducted a user study highlighting that our visualizations are easy to use and useful as perceived by process mining experts. In this way, our work contributes to research on process mining, event sequence analysis, and visualization of temporal data.

Supporting Visual Exploration of Iterative Job Scheduling

We consider the general problem known as job shop scheduling, in which multiple jobs consist of sequential operations that need to be executed or served by appropriate machines having limited capacities. For example, train journeys (jobs) consist of moves and stops (operations) to be served by rail tracks and stations (machines). A schedule is an assignment of the job operations to machines and times where and when they will be executed. The developers of computational methods for job scheduling need tools enabling them to explore how their methods work. At a high level of generality, we define the system of pertinent exploration tasks and a combination of visualizations capable of supporting the tasks. We provide general descriptions of the purposes, contents, visual encoding, properties, and interactive facilities of the visualizations and illustrate them with images from an example implementation in air traffic management. We justify the design of the visualizations based on the tasks, principles of creating visualizations for pattern discovery, and scalability requirements. The outcomes of our research are sufficiently general to be of use in a variety of applications.

Situated Visual Analysis and Live Monitoring for Manufacturing

Modern machines continuously log status reports over long periods of time, which are valuable data to optimize working routines. Data visualization is a commonly used tool to gain insights into these data, mostly in retrospective (e.g., to determine causal dependencies between the faults of different machines). We present an approach to bring such visual analyses to the shop floor to support reacting to faults in real time. This approach combines spatio-temporal analyses of time series using a handheld touch device with augmented reality for live monitoring. Important information augments machines directly in their real-world context, and detailed logs of current and historical events are displayed on the handheld device. In collaboration with an industry partner, we designed and tested our approach on a live production line to obtain feedback from operators. We compare our approach for monitoring and analysis with existing solutions that are currently deployed.

Reflections on Visualization Research Projects in the Manufacturing Industry

The rise of Industry 4.0 and cyber-physical systems has led to an abundance of large amounts of data, particularly in the manufacturing industry. Visualization and visual analytics play essential roles in harnessing this data. They have already been acknowledged as being among the key enabling technologies in the fourth industrial revolution. However, there are many challenges attached to applying visualization successfully, both from the manufacturing industry and visualization research perspectives. As members of research institutions involved in several applied research projects dealing with visualization in manufacturing, we characterized and analyzed our experiences for a detailed qualitative view, to distill important lessons learned, and to identify research gaps. With this article, we aim to provide added value and guidance for both manufacturing engineers and visualization researchers to avoid pitfalls and make such interdisciplinary endeavors more successful.

ManEx: The Visual Analysis of Measurements for the Assessment of Errors in Electrical Engines

Electrical engines are a key technology all automotive manufacturers must master to stay competitive. Engineers need to analyze an overwhelming number of engine measurements to improve the manufacturing for this technology. They are hindered in the task of analyzing large numbers of engines, however, by the following challenges: 1) Engines comprise a complex hierarchical structure of subcomponents. 2) Locating the cause of errors along manufacturing processes is a difficult procedure. 3) Large numbers of heterogeneous measurements impair the ability to explain errors in engines. We address these challenges in a design study with automotive engineers and by developing the visual analytics system Manufacturing Explorer (ManEx), which provides interactive interfaces to analyze measurements of engines across the manufacturing process. ManEx was validated by five experts. Our results suggest high usability and usefulness scores and the improvement of a real-world manufacturing process. Specifically, with ManEx, experts reduced scraped parts by over 3%.

A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?

It has been widely suggested that a key goal of visualization systems is to assist decision making, but is this true? We conduct a critical investigation on whether the activity of decision making is indeed central to the visualization domain. By approaching decision making as a user task, we explore the degree to which decision tasks are evident in visualization research and user studies. Our analysis suggests that decision tasks are not commonly found in current visualization task taxonomies and that the visualization field has yet to leverage guidance from decision theory domains on how to study such tasks. We further found that the majority of visualizations addressing decision making were not evaluated based on their ability to assist decision tasks. Finally, to help expand the impact of visual analytics in organizational as well as casual decision making activities, we initiate a research agenda on how decision making assistance could be elevated throughout visualization research.

A Visualization Approach for Monitoring Order Processing in E-Commerce Warehouse

The efficiency of warehouses is vital to e-commerce. Fast order processing at the warehouses ensures timely deliveries and improves customer satisfaction. However, monitoring, analyzing, and manipulating order processing in the warehouses in real time are challenging for traditional methods due to the sheer volume of incoming orders, the fuzzy definition of delayed order patterns, and the complex decision-making of order handling priorities. In this paper, we adopt a data-driven approach and propose OrderMonitor, a visual analytics system that assists warehouse managers in analyzing and improving order processing efficiency in real time based on streaming warehouse event data. Specifically, the order processing pipeline is visualized with a novel pipeline design based on the sedimentation metaphor to facilitate real-time order monitoring and suggest potentially abnormal orders. We also design a novel visualization that depicts order timelines based on the Gantt charts and Marey's graphs. Such a visualization helps the managers gain insights into the performance of order processing and find major blockers for delayed orders. Furthermore, an evaluating view is provided to assist users in inspecting order details and assigning priorities to improve the processing performance. The effectiveness of OrderMonitor is evaluated with two case studies on a real-world warehouse dataset.

OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples

One major cause of performance degradation in predictive models is that the test samples are not well covered by the training data. Such not well-represented samples are called OoD samples. In this article, we propose OoDAnalyzer, a visual analysis approach for interactively identifying OoD samples and explaining them in context. Our approach integrates an ensemble OoD detection method and a grid-based visualization. The detection method is improved from deep ensembles by combining more features with algorithms in the same family. To better analyze and understand the OoD samples in context, we have developed a novel kNN-based grid layout algorithm motivated by Hall's theorem. The algorithm approximates the optimal layout and has O(kN²) time complexity, faster than the grid layout algorithm with overall best performance but O(N³) time complexity. Quantitative evaluation and case studies were performed on several datasets to demonstrate the effectiveness and usefulness of OoDAnalyzer.

A Visual Analytics Framework for Reviewing Multivariate Time-Series Data with Dimensionality Reduction

Data-driven problem solving in many real-world applications involves analysis of time-dependent multivariate data, for which dimensionality reduction (DR) methods are often used to uncover the intrinsic structure and features of the data. However, DR is usually applied to a subset of data that is either single-time-point multivariate or univariate time-series, resulting in the need to manually examine and correlate the DR results out of different data subsets. When the number of dimensions is large either in terms of the number of time points or attributes, this manual task becomes too tedious and infeasible. In this paper, we present MulTiDR, a new DR framework that enables processing of time-dependent multivariate data as a whole to provide a comprehensive overview of the data. With the framework, we employ DR in two steps. When treating the instances, time points, and attributes of the data as a 3D array, the first DR step reduces the three axes of the array to two, and the second DR step visualizes the data in a lower-dimensional space. In addition, by coupling with a contrastive learning method and interactive visualizations, our framework enhances analysts' ability to interpret DR results. We demonstrate the effectiveness of our framework with four case studies using real-world datasets.

A Systematic Review of Visualization in Building Information Modeling

Building Information Modeling (BIM) employs data-rich 3D CAD models for large-scale facility design, construction, and operation. These complex datasets contain a large amount and variety of information, ranging from design specifications to real-time sensor data. They are used by architects and engineers for various analysis and simulations throughout a facility's life cycle. Many techniques from different visualization fields could be used to analyze these data. However, the BIM domain still remains largely unexplored by the visualization community. The goal of this article is to encourage visualization researchers to increase their involvement with BIM. To this end, we present the results of a systematic review of visualization in current BIM practice. We use a novel taxonomy to identify main application areas and analyze commonly employed techniques. From this domain characterization, we highlight future research opportunities brought forth by the unique features of BIM. For instance, exploring the synergies between scientific and information visualization to integrate spatial and non-spatial data. We hope this article raises awareness to interesting new challenges the BIM domain brings to the visualization community.

Visual Analytics for Electromagnetic Situation Awareness in Radio Monitoring and Management

Traditional radio monitoring and management largely depend on radio spectrum data analysis, which requires considerable domain experience and heavy cognition effort and frequently results in incorrect signal judgment and incomprehensive situation awareness. Faced with increasingly complicated electromagnetic environments, radio supervisors urgently need additional data sources and advanced analytical technologies to enhance their situation awareness ability. This paper introduces a visual analytics approach for electromagnetic situation awareness. Guided by a detailed scenario and requirement analysis, we first propose a signal clustering method to process radio signal data and a situation assessment model to obtain qualitative and quantitative descriptions of the electromagnetic situations. We then design a two-module interface with a set of visualization views and interactions to help radio supervisors perceive and understand the electromagnetic situations by a joint analysis of radio signal data and radio spectrum data. Evaluations on real-world data sets and an interview with actual users demonstrate the effectiveness of our prototype system. Finally, we discuss the limitations of the proposed approach and provide future work directions.

An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data

Dimensionality reduction (DR) methods are commonly used for analyzing and visualizing multidimensional data. However, when data is a live streaming feed, conventional DR methods cannot be directly used because of their computational complexity and inability to preserve the projected data positions at previous time points. In addition, the problem becomes even more challenging when the dynamic data records have a varying number of dimensions as often found in real-world applications. This paper presents an incremental DR solution. We enhance an existing incremental PCA method in several ways to ensure its usability for visualizing streaming multidimensional data. First, we use geometric transformation and animation methods to help preserve a viewer's mental map when visualizing the incremental results. Second, to handle data dimension variants, we use an optimization method to estimate the projected data positions, and also convey the resulting uncertainty in the visualization. We demonstrate the effectiveness of our design with two case studies using real-world datasets.

PlanningVis: A Visual Analytics Approach to Production Planning in Smart Factories

Production planning in the manufacturing industry is crucial for fully utilizing factory resources (e.g., machines, raw materials and workers) and reducing costs. With the advent of industry 4.0, plenty of data recording the status of factory resources have been collected and further involved in production planning, which brings an unprecedented opportunity to understand, evaluate and adjust complex production plans through a data-driven approach. However, developing a systematic analytics approach for production planning is challenging due to the large volume of production data, the complex dependency between products, and unexpected changes in the market and the plant. Previous studies only provide summarized results and fail to show details for comparative analysis of production plans. Besides, the rapid adjustment to the plan in the case of an unanticipated incident is also not supported. In this paper, we propose PlanningVis, a visual analytics system to support the exploration and comparison of production plans with three levels of details: a plan overview presenting the overall difference between plans, a product view visualizing various properties of individual products, and a production detail view displaying the product dependency and the daily production details in related factories. By integrating an automatic planning algorithm with interactive visual explorations, PlanningVis can facilitate the efficient optimization of daily production planning as well as support a quick response to unanticipated incidents in manufacturing. Two case studies with real-world data and carefully designed interviews with domain experts demonstrate the effectiveness and usability of PlanningVis.

A Visual Analysis Approach for Understanding Durability Test Data of Automotive Products

People face data-rich manufacturing environments in Industry 4.0. As an important technology for explaining and understanding complex data, visual analytics has been increasingly introduced into industrial data analysis scenarios. With the durability test of automotive starters as background, this study proposes a visual analysis approach for understanding large-scale and long-term durability test data. Guided by detailed scenario and requirement analyses, we first propose a migration-adapted clustering algorithm that utilizes a segmentation strategy and a group of matching-updating operations to achieve an efficient and accurate clustering analysis of the data for starting mode identification and abnormal test detection. We then design and implement a visual analysis system that provides a set of user-friendly visual designs and lightweight interactions to help people gain data insights into the test process overview, test data patterns, and durability performance dynamics. Finally, we conduct a quantitative algorithm evaluation, case study, and user interview by using real-world starter durability test datasets. The results demonstrate the effectiveness of the approach and its possible inspiration for the durability test data analysis of other similar industrial products.

Where do we start? Guidance for technology implementation in maintenance management for manufacturing

Recent efforts in Smart Manufacturing (SM) have proven quite effective at elucidating system behavior using sensing systems, communications and computational platforms, along with statistical methods to collect and analyze real-time performance data. However, how do you effectively select where and when to implement these technology solutions within manufacturing operations? Furthermore, how do you account for the human-driven activities in manufacturing when inserting new technologies? Due to a reliance on human problem solving skills, today's maintenance operations are largely manual processes without wide-spread automation. The current state-of-the-art maintenance management systems and out-of-the-box solutions do not directly provide necessary synergy between human and technology, and many paradigms ultimately keep the human and digital knowledge systems separate. Decision makers are using one or the other on a case-by-case basis, causing both human and machine to cannibalize each other's function, leaving both disadvantaged despite ultimately having common goals. A new paradigm can be achieved through a hybridized systems approach - where human intelligence is effectively augmented with sensing technology and decision support tools, including analytics, diagnostics, or prognostic tools. While these tools promise more efficient, cost-effective maintenance decisions, and improved system productivity, their use is hindered when it is unclear what core organizational or cultural problems they are being implemented to solve. To explicitly frame our discussion about implementation of new technologies in maintenance management around these problems, we adopt well established error mitigation frameworks from human factors experts - who have promoted human-systems integration for decades - to maintenance in manufacturing. Our resulting tiered mitigation strategy guides where and how to insert SM technologies into a human-dominated maintenance management process.

AirInsight: Visual Exploration and Interpretation of Latent Patterns and Anomalies in Air Quality Data

Nowadays, huge volume of air quality data provides unprecedented opportunities for analyzing pollution. However, due to the high complexity, most traditional analytical methods focus on abstracting data, so these techniques discard the original structure and limit the understanding of the results. Visual analysis is a powerful technique for exploring unknown patterns since it retains the details of the original data and gives visual feedback to users. In this paper, we focus on air quality data and propose the AirInsight design, an interactive visual analytic system for recognizing, exploring, and summarizing regular patterns, as well as detecting, classifying, and interpreting abnormal cases. Based on the time-varying and multivariate features of air quality data, a dimension reduction method Composite Least Square Projection (CLSP) is proposed, which allows appreciating and interpreting the data patterns in the context of attributes. On the basis of the observed regular patterns, multiple abnormal cases are further detected, including the multivariate anomalies by the proposed Noise Hierarchical Clustering (NHC) method, abruptly changing timestamps by Time diversity (TD) indicator, and cities with unique patterns by the Geographical Surprise (GS) measure. Moreover, we combine TD and GS to group anomalies based on their underlying spatiotemporal correlations. AirInsight includes multiple coordinated views and rich interactive functions to provide contextual information from different aspects and facilitate a comprehensive understanding. In particular, a pair of glyphs are designed that provide a visual representation of the temporal variation in air quality conditions for a user-selected city. Experiments show that CLSP improves the accuracy of Least Square Projection (LSP) and that NHC has the ability to separate noises. Meanwhile, several case studies and task-based user evaluation demonstrate that our system is effective and practical for exploring and interpreting multivariate spatiotemporal patterns and anomalies in air quality data.

A Visual Analytics Framework for the Detection of Anomalous Call Stack Trees in High Performance Computing Applications

Anomalous runtime behavior detection is one of the most important tasks for performance diagnosis in High Performance Computing (HPC). Most of the existing methods find anomalous executions based on the properties of individual functions, such as execution time. However, it is insufficient to identify abnormal behavior without taking into account the context of the executions, such as the invocations of children functions and the communications with other HPC nodes. We improve upon the existing anomaly detection approaches by utilizing the call stack structures of the executions, which record rich temporal and contextual information. With our call stack tree (CSTree) representation of the executions, we formulate the anomaly detection problem as finding anomalous tree structures in a call stack forest. The CSTrees are converted to vector representations using our proposed stack2vec embedding. Structural and temporal visualizations of CSTrees are provided to support users in the identification and verification of the anomalies during an active anomaly detection process. Three case studies of real-world HPC applications demonstrate the capabilities of our approach.

Analyzing the Training Processes of Deep Generative Models

Among the many types of deep models, deep generative models (DGMs) provide a solution to the important problem of unsupervised and semi-supervised learning. However, training DGMs requires more skill, experience, and know-how because their training is more complex than other types of deep models such as convolutional neural networks (CNNs). We develop a visual analytics approach for better understanding and diagnosing the training process of a DGM. To help experts understand the overall training process, we first extract a large amount of time series data that represents training dynamics (e.g., activation changes over time). A blue-noise polyline sampling scheme is then introduced to select time series samples, which can both preserve outliers and reduce visual clutter. To further investigate the root cause of a failed training process, we propose a credit assignment algorithm that indicates how other neurons contribute to the output of the neuron causing the training failure. Two case studies are conducted with machine learning experts to demonstrate how our approach helps understand and diagnose the training processes of DGMs. We also show how our approach can be directly used to analyze other types of deep models, such as CNNs.

GENERATING CONTEXTUAL DESIGN FOR ENVIRONMENT PRINCIPLES IN SUSTAINABLE MANUFACTURING USING VISUAL ANALYTICS

Design for Environment (DfE) principles are helpful for integrating manufacturing-specific environmental sustainability considerations into product and process design. However, such principles are often overly general, static, and disconnected from production contexts. This paper proposes a visual analytics-based framework for generating DfE principles that are contextualized to specific production setups. These principles are generated through interactive visual exploration of design and process parameters as well as manufacturing process performance metrics corresponding to the production setup. We also develop a formal schema for aiding storage, updating, and reuse of the generated DfE principles. In this schema, each DfE principle is associated with corresponding product lifecycle data and the evidence that led to the generation of that principle. We demonstrate the proposed visual analytics framework using data from an industry-led experiment that compared dry ice based and oil based milling for a specific production setup.

Visual Analytics Tools for Sustainable Lifecycle Design: Current Status, Challenges, and Future Opportunities

The rapid rise in technologies for data collection has created an unmatched opportunity to advance the use of data-rich tools for lifecycle decision-making. However, the usefulness of these technologies is limited by the ability to translate lifecycle data into actionable insights for human decision-makers. This is especially true in the case of sustainable lifecycle design (SLD), as the assessment of environmental impacts, and the feasibility of making corresponding design changes, often relies on human expertise and intuition. Supporting human sense-making in SLD requires the use of both data-driven and user-driven methods while exploring lifecycle data. A promising approach for combining the two is through the use of visual analytics (VA) tools. Such tools can leverage the ability of computer-based tools to gather, process, and summarize data along with the ability of human-experts to guide analyses through domain knowledge or data-driven insight. In this paper, we review previous research that has created VA tools in SLD. We also highlight existing challenges and future opportunities for such tools in different lifecycle stages-design, manufacturing, distribution & supply chain, use-phase, end-of-life, as well as life cycle assessment. Our review shows that while the number of VA tools in SLD is relatively small, researchers are increasingly focusing on the subject matter. Our review also suggests that VA tools can address existing challenges in SLD and that significant future opportunities exist.