Infusing Expert Knowledge Into a Deep Neural Network Using Attention Mechanism for Personalized Learning Environments

Machine learning models are biased toward data seen during the training steps. The models will tend to give good results in classes where there are many examples and poor results in those with few examples. This problem generally occurs when the classes to predict are imbalanced and this is frequent in educational data where for example, there are skills that are very difficult or very easy to master. There will be less data on students that correctly answered questions related to difficult skills and who incorrectly answered those related to skills easy to master. In this paper, we tackled this problem by proposing a hybrid architecture combining Deep Neural Network architectures— especially Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNN)—with expert knowledge for user modeling. The proposed solution uses attention mechanism to infuse expert knowledge into the Deep Neural Network. It has been tested in two contexts: knowledge tracing in an intelligent tutoring system (ITS) called Logic-Muse and prediction of socio-moral reasoning in a serious game called MorALERT. The proposed solution is compared to state-of-the-art machine learning solutions and experiments show that the resulting model can accurately predict the current student's knowledge state (in Logic-Muse) and thus enable an accurate personalization of the learning process. Other experiments show that the model can also be used to predict the level of socio-moral reasoning skills (in MorALERT). Our findings suggest the need for hybrid neural networks that integrate prior expert knowledge (especially when it is necessary to compensate for the strong dependency—of deep learning methods—on data size or the possible unbalanced datasets). Many domains can benefit from such an approach to building models that allow generalization even when there are small training data.

Developing a model to evaluate and improve user experience with hand motions in virtual reality environments

In video games, the evaluation of the user experience (UX) mainly refers to two main groups of aspects, those that refer to the player that is mainly oriented to make the player feel good while playing and those that refer to the video game that is oriented to make the video game easy to understand and play. The aspects considered that are related to the player are engagement, enjoyment, and flow; the aspects related to video game, usability, and dependability. Virtual reality environments today have changed the paradigm in various fields of application, such as health, education, entertainment, among others. Therefore, it is important to observe the effects of handedness with hand movements in virtual reality environments. This work proposes a model to evaluate and improve the user experience considering player and video game aspects, taking into account handedness with hand movements in virtual reality environments. Player and video game aspects can be added to evaluations of the effect of handedness, especially in virtual reality environments, in order to know the user's behavior in terms of skill, performance, and accuracy, among other features by using a particular hand to perform specific tasks. Next, a case study is presented with two groups of users using a virtual reality environment to perform several user tasks considering the dominant and non-dominant hand. By evaluating the user tasks it is possible to know the levels of engagement, enjoyment, motivation, and usability in a virtual reality environment. Finally, an analysis of results is presented in which several improvements of UX are presented.

Effects of AR Picture Books on German Teaching in Universities

In this paper, we discuss the teaching effects of augmented reality (AR) technology in German instruction. We conducted one prestudy and three formal studies on German learners in China’s mainland and Taiwan region. In the formal studies, a total of 120 students participated in the survey, allowing us to compare the differences in interest in learning between AR picture books and traditional picture books. A total of 114 students took part in the survey, which enabled us to compare the contribution of AR picture books to teaching when students’ satisfaction and German proficiency were different. To improve satisfaction, 514 students participated in the survey regarding the influence of the interactive narrative design effect and peer learning on satisfaction with using AR picture books. The results suggest that when learning German with AR picture books, satisfaction is the key construct that determines students’ learning states.

A systematic approach for English education model based on the neural network algorithm

This paper algorithms based on neural network model designed for English education, to develop a model education system with artificial intelligence, summarized the dimensions were can be used for data analysis related indicators. These indicators include not only the contents of the learning behavior, test behavior, cooperation behavior and resource search behavior and other human-computer interaction behavior data, also includes demographic background information, learning ability, learning attitude, and other characteristic data that affect the learning effect. We tried to collect relevant indicators to the maximum extent. An audiovisual fusion method based on Convolutional Neural Network (CNN) is proposed. The independent CNN structure is used to realize independent modeling of audiovisual perception and asynchronous information transmission and obtain the description of audiovisual parallel data in the high-dimensional feature space. Following the shared fully connected structure, it is possible to model the long-term dependence of audiovisual parallel data in a higher dimension. Experiments show that the AVSR system built using a CNN-based audiovisual fusion method can achieve a significant performance improvement, and its recognition error rate is relatively reduced by about 15%. The speech recognition system trained with the cross-domain adaptive method can obtain a significant performance improvement, and its recognition error rate is more than 10% lower than that of the baseline system..

Engagement Enhancement Based on Human-in-the-Loop Optimization for Neural Rehabilitation

Enhancing patients' engagement is of great benefit for neural rehabilitation. However, physiological and neurological differences among individuals can cause divergent responses to the same task, and the responses can further change considerably during training; both of these factors make engagement enhancement a challenge. This challenge can be overcome by training task optimization based on subjects' responses. To this end, an engagement enhancement method based on human-in-the-loop optimization is proposed in this paper. Firstly, an interactive speed-tracking riding game is designed as the training task in which four reference speed curves (RSCs) are designed to construct the reference trajectory in each generation. Each RSC is modeled using a piecewise function, which is determined by the starting velocity, transient time, and end velocity. Based on the parameterized model, the difficulty of the training task, which is a key factor affecting the engagement, can be optimized. Then, the objective function is designed with consideration to the tracking accuracy and the surface electromyogram (sEMG)-based muscle activation, and the physical and physiological responses of the subjects can consequently be evaluated simultaneously. Moreover, a covariance matrix adaption evolution strategy, which is relatively tolerant of both measurement noises and human adaptation, is used to generate the optimal parameters of the RSCs periodically. By optimization of the RSCs persistently, the objective function can be maximized, and the subjects' engagement can be enhanced. Finally, the performance of the proposed method is demonstrated by the validation and comparison experiments. The results show that both subjects' sEMG-based motor engagement and electroencephalography based neural engagement can be improved significantly and maintained at a high level.

Human Emotion Recognition: Review of Sensors and Methods

Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.

Classification of Multiple Psychological Dimensions in Computer Game Players Using Physiology, Performance, and Personality Characteristics

Human psychological (cognitive and affective) dimensions can be assessed using several methods, such as physiological or performance measurements. To date, however, few studies have compared different data modalities with regard to their ability to enable accurate classification of different psychological dimensions. This study thus compares classification accuracies for four psychological dimensions and two subjective preferences about computer game difficulty using three data modalities: physiology, performance, and personality characteristics. Thirty participants played a computer game at nine difficulty configurations that were implemented via two difficulty parameters. In each configuration, seven physiological measurements and two performance variables were recorded. A short questionnaire was filled out to assess the perceived difficulty, enjoyment, valence, arousal, and the way the participant would like to modify the two difficulty parameters. Furthermore, participants’ personality characteristics were assessed using four questionnaires. All combinations of the three data modalities (physiology, performance, and personality) were used to classify six dimensions of the short questionnaire into either two, three or many classes using four classifier types: linear discriminant analysis, support vector machine (SVM), ensemble decision tree, and multiple linear regression. The classification accuracy varied widely between the different psychological dimensions; the highest accuracies for two-class and three-class classification were 97.6 and 84.1%, respectively. Normalized physiological measurements were the most informative data modality, though current game difficulty, personality and performance also contributed to classification accuracy; the best selected features are presented and discussed in the text. The SVM and multiple linear regression were the most accurate classifiers, with regression being more effective for normalized physiological data. In the future, we will further examine the effect of different classification approaches on user experience by detecting the user’s psychological state and adapting game difficulty in real-time. This will allow us to obtain a complete picture of the performance of affect-aware systems in both an offline (classification accuracy) and real-time (effect on user experience) fashion.

Evolutionary Search in the Space of Rules for Creation of New Two-Player Board Games

Games have always been a popular test bed for artificial intelligence techniques. Game developers are always in constant search for techniques that can automatically create computer games minimizing the developer's task. In this work we present an evolutionary strategy based solution towards the automatic generation of two player board games. To guide the evolutionary process towards games, which are entertaining, we propose a set of metrics. These metrics are based upon different theories of entertainment in computer games. This work also compares the entertainment value of the evolved games with the existing popular board based games. Further to verify the entertainment value of the evolved games with the entertainment value of the human user a human user survey is conducted. In addition to the user survey we check the learnability of the evolved games using an artificial neural network based controller. The proposed metrics and the evolutionary process can be employed for generating new and entertaining board games, provided an initial search space is given to the evolutionary algorithm.