Predicting Affect from Gaze Data during Interaction with an Intelligent Tutoring System

Automatic modeling of student characteristics with interaction and physiological data using machine learning: A review

Student characteristics affect their willingness and ability to acquire new knowledge. Assessing and identifying the effects of student characteristics is important for online educational systems. Machine learning (ML) is becoming significant in utilizing learning data for student modeling, decision support systems, adaptive systems, and evaluation systems. The growing need for dynamic assessment of student characteristics in online educational systems has led to application of machine learning methods in modeling the characteristics. Being able to automatically model student characteristics during learning processes is essential for dynamic and continuous adaptation of teaching and learning to each student's needs. This paper provides a review of 8 years (from 2015 to 2022) of literature on the application of machine learning methods for automatic modeling of various student characteristics. The review found six student characteristics that can be modeled automatically and highlighted the data types, collection methods, and machine learning techniques used to model them. Researchers, educators, and online educational systems designers will benefit from this study as it could be used as a guide for decision-making when creating student models for adaptive educational systems. Such systems can detect students' needs during the learning process and adapt the learning interventions based on the detected needs. Moreover, the study revealed the progress made in the application of machine learning for automatic modeling of student characteristics and suggested new future research directions for the field. Therefore, machine learning researchers could benefit from this study as they can further advance this area by investigating new, unexplored techniques and find new ways to improve the accuracy of the created student models.

A Scanner Deeply: Predicting Gaze Heatmaps On Visualizations Using Crowdsourced Eye Movement Data

Visual perception is a key component of data visualization. Much prior empirical work uses eye movement as a proxy to understand human visual perception. Diverse apparatus and techniques have been proposed to collect eye movements, but there is still no optimal approach. In this paper, we review 30 prior works for collecting eye movements based on three axes: (1) the tracker technology used to measure eye movements; (2) the image stimulus shown to participants; and (3) the collection methodology used to gather the data. Based on this taxonomy, we employ a webcam-based eyetracking approach using task-specific visualizations as the stimulus. The low technology requirement means that virtually anyone can participate, thus enabling us to collect data at large scale using crowdsourcing: approximately 12,000 samples in total. Choosing visualization images as stimulus means that the eye movements will be specific to perceptual tasks associated with visualization. We use these data to propose a SCANNER DEEPLY, a virtual eyetracker model that, given an image of a visualization, generates a gaze heatmap for that image. We employ a computationally efficient, yet powerful convolutional neural network for our model. We compare the results of our work with results from the DVS model and a neural network trained on the Salicon dataset. The analysis of our gaze patterns enables us to understand how users grasp the structure of visualized data. We also make our stimulus dataset of visualization images available as part of this paper's contribution.

Feedback beyond accuracy: Using eye‐tracking to detect comprehensibility and interest during reading

Knowing what information a user wants is a paramount challenge to information science and technology. Implicit feedback is key to solving this challenge, as it allows information systems to learn about a user's needs and preferences. The available feedback, however, tends to be limited and its interpretation shows to be difficult. To tackle this challenge, we present a user study that explores whether tracking the eyes can unpack part of the complexity inherent to relevance and relevance decisions. The eye behavior of 30 participants reading 18 news articles was compared with their subjectively appraised comprehensibility and interest at a discourse level. Using linear regression models, the eye-tracking signal explained 49.93% (comprehensibility) and 30.41% (interest) of variance (p < .001). We conclude that eye behavior provides implicit feedback beyond accuracy that enables new forms of adaptation and interaction support for personalized information systems.

Gaze-Driven Adaptive Interventions for Magazine-Style Narrative Visualizations

In this article, we investigate the value of gaze-driven adaptive interventions to support the processing of textual documents with embedded visualizations, i.e., Magazine Style Narrative Visualizations (MSNVs). These interventions are provided dynamically by highlighting relevant data points in the visualization when the user reads related sentences in the MSNV text, as detected by an eye-tracker. We conducted a user study during which participants read a set of MSNVs with our interventions, and compared their performance and experience with participants who received no interventions. Our work extends previous findings by showing that dynamic, gaze-driven interventions can be delivered based on reading behaviors in MSNVs, a widespread form of documents that have never been considered for gaze-driven adaptation so far. Next, we found that the interventions significantly improved the performance of users with low levels of visualization literacy, i.e., those users who need help the most due to their lower ability to process and understand data visualizations. However, high literacy users were not impacted by the interventions, providing initial evidence that gaze-driven interventions can be further improved by personalizing them to the levels of visualization literacy of their users.

Comparing and Combining Interaction Data and Eye-tracking Data for the Real-time Prediction of User Cognitive Abilities in Visualization Tasks

Previous work has shown that some user cognitive abilities relevant for processing information visualizations can be predicted from eye-tracking data. Performing this type of user modeling is important for devising visualizations that can detect a user's abilities and adapt accordingly during the interaction. In this article, we extend previous user modeling work by investigating for the first time interaction data as an alternative source to predict cognitive abilities during visualization processing when it is not feasible to collect eye-tracking data. We present an extensive comparison of user models based solely on eye-tracking data, on interaction data, as well as on a combination of the two. Although we found that eye-tracking data generate the most accurate predictions, results show that interaction data can still outperform a majority-class baseline, meaning that adaptation for interactive visualizations could be enabled even when it is not feasible to perform eye tracking, using solely interaction data. Furthermore, we found that interaction data can predict several cognitive abilities with better accuracy at the very beginning of the task than eye-tracking data, which are valuable for delivering adaptation early in the task. We also extend previous work by examining the value of multimodal classifiers combining interaction data and eye-tracking data, with promising results for some of our target user cognitive abilities. Next, we contribute to previous work by extending the type of visualizations considered and the set of cognitive abilities that can be predicted from either eye-tracking data and interaction data. Finally, we evaluate how noise in gaze data impacts prediction accuracy and find that retaining rather noisy gaze datapoints can yield equal or even better predictions than discarding them, a novel and important contribution for devising adaptive visualizations in real settings where eye-tracking data are typically noisier than in laboratory settings.

An Exploration of Machine Learning Methods for Robust Boredom Classification Using EEG and GSR Data

In recent years, affective computing has been actively researched to provide a higher level of emotion-awareness. Numerous studies have been conducted to detect the user's emotions from physiological data. Among a myriad of target emotions, boredom, in particular, has been suggested to cause not only medical issues but also challenges in various facets of daily life. However, to the best of our knowledge, no previous studies have used electroencephalography (EEG) and galvanic skin response (GSR) together for boredom classification, although these data have potential features for emotion classification. To investigate the combined effect of these features on boredom classification, we collected EEG and GSR data from 28 participants using off-the-shelf sensors. During data acquisition, we used a set of stimuli comprising a video clip designed to elicit boredom and two other video clips of entertaining content. The collected samples were labeled based on the participants' questionnaire-based testimonies on experienced boredom levels. Using the collected data, we initially trained 30 models with 19 machine learning algorithms and selected the top three candidate classifiers. After tuning the hyperparameters, we validated the final models through 1000 iterations of 10-fold cross validation to increase the robustness of the test results. Our results indicated that a Multilayer Perceptron model performed the best with a mean accuracy of 79.98% (AUC: 0.781). It also revealed the correlation between boredom and the combined features of EEG and GSR. These results can be useful for building accurate affective computing systems and understanding the physiological properties of boredom.

Development of a support system for measurement and analysis of thinking processes based on a metacognitive interpretation framework: a case study of dissolution of belief conflict thinking processes

The ability for metacognitive thought, or "thinking about thinking," is recognized as an increasingly important skill for the future enrichment of social life. However, this skill is difficult to teach because it involves implicit cognitive activities that cannot be perceived by an outside observer. In this study, we propose an interpretation framework of metacognition as one approach to considering metacognitive thinking processes. This framework serves as the design principles for developing a system that makes it possible to provide metacognitive interpretations of gaze behaviors and thought operation actions and provides a common basis for sharing and comparing knowledge from analysis results. In this study, for an example of framework-based system development, we construct a thinking externalization application and thinking analysis support system with a thinking task of dissolving belief conflict as the theme. We also demonstrate an example of the analysis of thinking about belief conflict, as derived from lower-level and higher-level thinking interpretation rules. From the example results of the defined interpretation rules, we found that the desired behavior occurred, demonstrating the postulated possibility of capturing the thought process. By realizing a series of phases on the framework proposed in this paper, it contributes to the feasibility of grasping the metacognition process and accumulating knowledge about it.

Toward a Feature-Driven Understanding of Students' Emotions during Interactions with Agent-Based Learning Environments

This selective review synthesizes and draws recommendations from the fields of affective computing, intelligent tutoring systems, and psychology to describe and discuss the emotions that learners report experiencing while interacting with agent-based learning environments (ABLEs). Theoretically driven explanations are provided that describe the relative effectiveness and ineffectiveness of different ABLE features to foster adaptive emotions (e.g., engagement, curiosity) vs. non-adaptive emotions (e.g., frustration, boredom) in six different environments. This review provides an analytical lens to evaluate and improve upon research with ABLEs by identifying specific system features and their relationship with learners' appraisals and emotions.