Capturing and Visualizing Provenance From Data Wrangling

IntiVisor: A Visual Analytics System for Interaction Log Analysis

Application developers frequently augment their code to produce event logs of specific operations performed by their users. Subsequent analysis of these event logs can help provide insight about the users' behavior relative to its intended use. The analysis process typically includes both event organization and pattern discovery activities. However, most existing visual analytics systems for interaction log analysis excel at supporting pattern discovery and overlook the importance of flexible event organization. This omission limits the practical application of these systems. Therefore, we developed a novel visual analytics system called IntiVisor that implements the entire end-to-end interaction analysis approach. An evaluation of the system with interaction data from four visualization applications showed the value and importance of supporting event organization in interaction log analysis.

Tracking provenance in clinical data warehouses for quality management

INTRODUCTION

Data provenance, which documents the origin, history, and transformations of data, can enhance the reproducibility of processing workflows and help to address errors and quality issues. In this work, we focus on tracking and utilizing provenance information as part of quality management in Extract-Transform-Load (ETL) processes used to build clinical data warehouses.

METHODS

We designed and implemented a framework that automatically tracks how data flows through an ETL process and detects errors and quality problems during processing. This information is then reported against an Application Programming Interface (API) that stores the issues along with contextual information on their location within the data being transformed and the overall workflow. We further designed a dashboard that supports health data engineers with inspecting the encountered issues and tracing them back to their root causes.

RESULTS

The framework was implemented in Java using the Spring Framework and integrated into ETL processes for Informatics for Integrating Biology and the Bedside (i2b2). The dashboard was realized using Grafana. We evaluated our approach on three different ETL processes for real-world datasets used to integrate them into our i2b2 clinical data warehouse. Using the provenance dashboard, we were able to identify frequent error patterns and link them to specific data points from the sources as well as ETL process steps. Provenance tracking increased the execution times of loading processes with an impact depending on the number of identified issues.

CONCLUSIONS

Provenance tracking can be a valuable tool for implementing continuous quality management for ETL processes. Relevant information can be collected from existing ETL workloads using dedicated APIs and visualized through dashboards, which support the identification of frequent patterns of problems together with their root causes, providing valuable information for improvements.

BEMTrace: Visualization-Driven Approach for Deriving Building Energy Models from BIM

Building Information Modeling (BIM) describes a central data pool covering the entire life cycle of a construction project. Similarly, Building Energy Modeling (BEM) describes the process of using a 3D representation of a building as a basis for thermal simulations to assess the building's energy performance. This paper explores the intersection of BIM and BEM, focusing on the challenges and methodologies in converting BIM data into BEM representations for energy performance analysis. BEMTrace integrates 3D data wrangling techniques with visualization methodologies to enhance the accuracy and traceability of the BIM-to-BEM conversion process. Through parsing, error detection, and algorithmic correction of BIM data, our methods generate valid BEM models suitable for energy simulation. Visualization techniques provide transparent insights into the conversion process, aiding error identification, validation, and user comprehension. We introduce context-adaptive selections to facilitate user interaction and to show that the BEMTrace workflow helps users understand complex 3D data wrangling processes.

Visual Analytics Meets Process Mining: Challenges and Opportunities

Visual analytics (VA) integrates the outstanding capabilities of humans in terms of visual information exploration with the enormous processing power of computers to form a powerful knowledge discovery environment. In other words, VA is the science of analytical reasoning facilitated by interactive interfaces, capturing the information discovery process while keeping humans in the loop. Process mining (PM) is a data-driven and process centric approach that aims to extract information and knowledge from event logs to discover, monitor, and improve processes in various application domains. The combination of interactive visual data analysis and exploration with PM algorithms can make complex information structures more comprehensible and facilitate new insights. Yet, this combination remains largely unexplored. In this article, we illustrate the concepts of VA and PM, how their combination can support the extraction of more insights from complex event data, and elaborate on the challenges and opportunities for analyzing process data with VA methods and enhancing VA methods using PM techniques.

Visualizing the Scripts of Data Wrangling With Somnus

Data workers use various scripting languages for data transformation, such as SAS, R, and Python. However, understanding intricate code pieces requires advanced programming skills, which hinders data workers from grasping the idea of data transformation at ease. Program visualization is beneficial for debugging and education and has the potential to illustrate transformations intuitively and interactively. In this article, we explore visualization design for demonstrating the semantics of code pieces in the context of data transformation. First, to depict individual data transformations, we structure a design space by two primary dimensions, i.e., key parameters to encode and possible visual channels to be mapped. Then, we derive a collection of 23 glyphs that visualize the semantics of transformations. Next, we design a pipeline, named Somnus, that provides an overview of the creation and evolution of data tables using a provenance graph. At the same time, it allows detailed investigation of individual transformations. User feedback on Somnus is positive. Our study participants achieved better accuracy with less time using Somnus, and preferred it over carefully-crafted textual description. Further, we provide two example applications to demonstrate the utility and versatility of Somnus.

Machine Learning Methods to Estimate Productivity of Harvesters: Mechanized Timber Harvesting in Brazil

The correct capture of forest operations information carried out in forest plantations can help in the management of mechanized harvesting timber. Proper management must be able to dimension resources and tools necessary for the fulfillment of operations and helping in strategic, tactical, and operational planning. In order to facilitate the decision making of forest managers, this work aimed to analyze the performance of machine learning algorithms in estimating the productivity of timber harvesters. As predictors of productivity, we used the availability of hours of machine use, individual mean volumes of trees, and terrain slopes. The dataset was composed of 144,973 records, carried out over a period of 28 months. We tested the predictive performance of 24 machine learning algorithms in default mode. In addition, we tested the performance of blending and stacking joint learning methods. We evaluated the model’s fit using the root mean squared error, mean absolute error, mean absolute percentage error, and determination coefficient. After cleaning the initial database, we used only 1.12% to build the model. Learning by blending ensemble stood out with a determination coefficient of 0.71 and a mean absolute percentage error of 15%. From the use of data from machine learning algorithms, it became possible to predict the productivity of timber harvesters. Testing a variety of machine learning algorithms with different dynamics contributed to the machine learning technique that helped us reach our goal: maximizing the model’s performance by conducting experimentation.