Real-time closed-loop control of cognitive load in neurological patients during robot-assisted gait training

Therapists and patients perceptions of a mixed reality system designed to improve trunk control and upper extremity function

A prototype system aimed at improving arm function and trunk control after stroke has been developed that combines mixed-reality (MR) feedback with a mobile seat system (Holoreach). The purpose of this study was to assess the usability of Holoreach in a rehabilitation setting from both the patient and therapist perspective. Ten therapists (eight physiotherapists and two occupational therapists) used the device in their regular therapy programs for fifteen stroke patients with trunk control issues. Each patient received four individual therapy sessions with the device performed under the supervision of the therapist. Therapists and patients kept therapy diaries and used customized questionnaires. At the end of the study two focus groups were conducted to further assess usability. Generally, the prototype system is suitable for training trunk and arm control. The therapists expressed overall positive views on the impact of Holoreach. They characterized it as new, motivating, fresh, joyful, interesting, and exciting. All therapists and 80% of the patients agreed with the statement that training with Holoreach is beneficial for rehabilitation. Nonetheless, improvements are required in the hardware and software, and design. The prototype system contributes at various levels to the rapidly evolving advances in neurorehabilitation, particularly regarding the practical aspect of exercise delivery.

Intensity control of robot-assisted gait training based on biometric data: Preliminary study

OBJECTIVE

This study is aimed to compare the effect of robot-assisted gait training when the intensity is controlled using patients' biometric data to when controlled by therapist's subjective judgment.

DESIGN

This is non-blinded, prospective, randomized controlled study. Patients were randomly assigned to one of two groups. In biometric data control group, exercise intensity was controlled through the patient's heart rate or rating of perceived exertion (RPE). The intensity was raised to the next level when the patient's heart rate reserve was less than 40 percent or the RPE was less than 12 points. The exercise intensity of the therapist control group was adjusted according to the judgement of a therapist. All patients were instructed to perform robot (Morning Walk®)-assisted 20-minute gait training session five times a week during 3 weeks. The primary outcome was functional ambulation category (FAC). The secondary outcomes were modified Barthel index (MBI), Berg balance scale (BBS), timed up and go test (TUG) and 10-meter walk test (10MWT) The outcomes were evaluated at baseline and after 3-week gait training.

RESULTS

A total of 55 patients with stroke were enrolled. After robotic rehabilitation, the primary outcome, FAC improved significantly (P < .05) in both groups. Also, secondary outcomes, including MBI, BBS, TUG, 10MWT, showed significant improvement (P < .05) in all groups. In addition, when comparing the functional change from baseline to week 3 between the two groups, there was no statistically significant difference in FAC (P > .05). The difference of baseline and week 3 of secondary outcome measure, MBI, BBS, TUG, 10MWT, showed no significant difference (P > .05).

CONCLUSION

In conclusion, when the robot intensity was adjusted using the patient's heart rate or RPE, the treatment effect has no significant difference to when adjusting the intensity according to the know-how of the therapist.

Gait Rehabilitation and Locomotion Support System Using a Distributed Controlled Robot System

In this paper, we proposed a distributed controlled gait rehabilitation and locomotion support system through human-robot cooperative control, and combination of two cane-type walking support robots (left and right) and one wheelchair robot to support the walking and locomotion of a person in need of walking assistance. The proposed system can realize five types of motion support from gait training to daily motion support, using three types of motion support modes with a distributed robot control system comprising up to three robots to support the user’s independence for walking and moving. The cane-type walking support robot moved in response to the manipulation force applied to the robot by the user, and can realize walking/movement support in all directions through the omnidirectional traveling part. In addition, the height of the robot can be adjusted according to the user’s physique, and the motion characteristics can be set according to the user’s walking ability. The wheelchair robot has a seat that can be raised, lowered, and tilted to provide standing assistance for the user and mobility support as an electric wheelchair. In this study, we developed a prototype of the proposed system and demonstrated its feasibility for five types of assistive actions in experiments with healthy subjects.

Challenges and solutions for application and wider adoption of wearable robots

The science and technology of wearable robots are steadily advancing, and the use of such robots in our everyday life appears to be within reach. Nevertheless, widespread adoption of wearable robots should not be taken for granted, especially since many recent attempts to bring them to real-life applications resulted in mixed outcomes. The aim of this article is to address the current challenges that are limiting the application and wider adoption of wearable robots that are typically worn over the human body. We categorized the challenges into mechanical layout, actuation, sensing, body interface, control, human-robot interfacing and coadaptation, and benchmarking. For each category, we discuss specific challenges and the rationale for why solving them is important, followed by an overview of relevant recent works. We conclude with an opinion that summarizes possible solutions that could contribute to the wider adoption of wearable robots.

Detecting Lies is a Child (Robot)’s Play: Gaze-Based Lie Detection in HRI

Robots destined to tasks like teaching or caregiving have to build a long-lasting social rapport with their human partners. This requires, from the robot side, to be capable of assessing whether the partner is trustworthy. To this aim a robot should be able to assess whether someone is lying or not, while preserving the pleasantness of the social interaction. We present an approach to promptly detect lies based on the pupil dilation, as intrinsic marker of the lie-associated cognitive load that can be applied in an ecological human–robot interaction, autonomously led by a robot. We demonstrated the validity of the approach with an experiment, in which the iCub humanoid robot engages the human partner by playing the role of a magician in a card game and detects in real-time the partner deceptive behavior. On top of that, we show how the robot can leverage on the gained knowledge about the deceptive behavior of each human partner, to better detect subsequent lies of that individual. Also, we explore whether machine learning models could improve lie detection performances for both known individuals (within-participants) over multiple interaction with the same partner, and with novel partners (between-participant). The proposed setup, interaction and models enable iCub to understand when its partners are lying, which is a fundamental skill for evaluating their trustworthiness and hence improving social human–robot interaction.

Measuring mental workload in assistive wearable devices: a review

As wearable assistive devices, such as prostheses and exoskeletons, become increasingly sophisticated and effective, the mental workload associated with their use remains high and becomes a major challenge to their ecological use and long-term adoption. Numerous methods of measuring mental workload co-exist, making analysis of this research topic difficult. The aim of this review is to examine how mental workload resulting from the use of wearable assistive devices has been measured, in order to gain insight into the specific possibilities and limitations of this field. Literature searches were conducted in the main scientific databases and 60 articles measuring the mental workload induced by the use of a wearable assistive device were included in this study. Three main families of methods were identified, the most common being 'dual task' and 'subjective assessment' methods, followed by those based on 'physiological measures', which included a wide variety of methods. The variability of the measurements was particularly high, making comparison difficult. There is as yet no evidence that any particular method of measuring mental workload is more appropriate to the field of wearable assistive devices. Each method has intrinsic limitations such as subjectivity, imprecision, robustness or complexity of implementation or interpretation. A promising metric seems to be the measurement of brain activity, as it is the only method that is directly related to mental workload. Finally, regardless of the measurement method chosen, special attention should be paid to the measurement of mental workload in the context of wearable assistive devices. In particular, certain practical considerations, such as ecological situations and environments or the level of expertise of the participants tested, may be essential to ensure the validity of the mental workload assessed.

Region-Level Functional and Effective Network Analysis of Human Brain During Cognitive Task Engagement

Mental disorders are a major source of disability, with few effective treatments. It has recently been argued that these diseases might be effectively treated by focusing on decision-making, and specifically remediating decision-making deficits that act as "ingredients" in these disorders. Prior work showed that direct electrical brain stimulation can enhance human cognitive control, and consequently decision-making. This raises a challenge of detecting cognitive control lapses directly from electrical brain activity. Here, we demonstrate approaches to overcome that challenge. We propose a novel method, referred to as maximal variance node merging (MVNM), that merges nodes within a brain region to construct informative inter-region brain networks. We employ this method to estimate functional (correlational) and effective (causal) networks using local field potentials (LFP) during a cognitive behavioral task. The effective networks computed using convergent cross mapping differentiate task engagement from background neural activity with 85% median classification accuracy. We also derive task engagement networks (TENs): networks that constitute the most discriminative inter-region connections. Subsequent graph analysis illustrates the crucial role of the dorsolateral prefrontal cortex (dlPFC) in task engagement, consistent with a widely accepted model for cognition. We also show that task engagement is linked to prefrontal cortex theta (4-8 Hz) oscillations. We, therefore, identify objective biomarkers associated with task engagement. These approaches may generalize to other cognitive functions, forming the basis of a network-based approach to detecting and rectifying decision deficits.

User Experience With Dynamic Difficulty Adjustment Methods for an Affective Exergame: Comparative Laboratory-Based Study

BACKGROUND

In affective exergames, game difficulty is dynamically adjusted to match the user's physical and psychological state. Such an adjustment is commonly made based on a combination of performance measures (eg, in-game scores) and physiological measurements, which provide insight into the player's psychological state. However, although many prototypes of affective games have been presented and many studies have shown that physiological measurements allow more accurate classification of the player's psychological state than performance measures, few studies have examined whether dynamic difficulty adjustment (DDA) based on physiological measurements (which requires additional sensors) results in a better user experience than performance-based DDA or manual difficulty adjustment.

OBJECTIVE

This study aims to compare five DDA methods in an affective exergame: manual (player-controlled), random, performance-based, personality-performance-based, and physiology-personality-performance-based (all-data).

METHODS

A total of 50 participants (N=50) were divided into five groups, corresponding to the five DDA methods. They played an exergame version of Pong for 18 minutes, starting at a medium difficulty; every 2 minutes, two game difficulty parameters (ball speed and paddle size) were adjusted using the participant's assigned DDA method. The DDA rules for the performance-based, personality-performance-based, and all-data groups were developed based on data from a previous open-loop study. Seven physiological responses were recorded throughout the sessions, and participants self-reported their preferred changes to difficulty every 2 minutes. After playing the game, participants reported their in-game experience using two questionnaires: the Intrinsic Motivation Inventory and the Flow Experience Measure.

RESULTS

Although the all-data method resulted in the most accurate changes to ball speed and paddle size (defined as the percentage match between DDA choice and participants' preference), no significant differences between DDA methods were found on the Intrinsic Motivation Inventory and Flow Experience Measure. When the data from all four automated DDA methods were pooled together, the accuracy of changes in ball speed was significantly correlated with players' enjoyment (r=0.38) and pressure (r=0.43).

CONCLUSIONS

Although our study is limited by the use of a between-subjects design and may not generalize to other exergame designs, the results do not currently support the inclusion of physiological measurements in affective exergames, as they did not result in an improved user experience. As the accuracy of difficulty changes is correlated with user experience, the results support the development of more effective DDA methods. However, they show that the inclusion of physiological measurements does not guarantee a better user experience even if it yields promising results in offline cross-validation.

Designing robot-assisted neurorehabilitation strategies for people with both HIV and stroke

There is increasing evidence that HIV is an independent risk factor for stroke, resulting in an emerging population of people living with both HIV and stroke all over the world. However, neurorehabilitation strategies for the HIV-stroke population are distinctly lacking, which poses an enormous global health challenge. In order to address this gap, a better understanding of the HIV-stroke population is needed, as well as potential approaches to design effective neurorehabilitation strategies for this population. This review goes into the mechanisms, manifestations, and treatment options of neurologic injury in stroke and HIV, the additional challenges posed by the HIV-stroke population, and rehabilitation engineering approaches for both high and low resource areas. The aim of this review is to connect the underlying neurologic properties in both HIV and stroke to rehabilitation engineering. It reviews what is currently known about the association between HIV and stroke and gaps in current treatment strategies for the HIV-stroke population. We highlight relevant current areas of research that can help advance neurorehabilitation strategies specifically for the HIV-stroke population. We then explore how robot-assisted rehabilitation combined with community-based rehabilitation could be used as a potential approach to meet the challenges posed by the HIV-stroke population. We include some of our own work exploring a community-based robotic rehabilitation exercise system. The most relevant strategies will be ones that not only take into account the individual status of the patient but also the cultural and economic considerations of their respective environment.

The effect of robot therapy assisted by surface EMG on hand recovery in post-stroke patients. A pilot study

Background: Hemiparesis caused by a stroke negatively limits a patient’s motor function. Nowadays, innovative technologies such as robots are commonly used in upper limb rehabilitation. The main goal of robot-aided therapy is to provide a maximum number of stimuli in order to stimulate brain neuroplasticity. Treatment applied in this study via the AMADEO robot aimed to improve finger flexion and extension. Aim: To assess the effect of rehabilitation assisted by a robot and enhanced by surface EMG. Research project: Before-after study design. Materials and methods: The study group consisted of 10 post-stroke patients enrolled for therapy with the AMADEO robot for at least 15 sessions. At the beginning and at the end of treatment, the following tests were used for clinical assessment: Fugl-Meyer scale, Box and Block test and Nine Hole Peg test. In the present study, we used surface electromyography (sEMG) to maintain optimal kinematics of hand motion. Whereas sensorial feedback, provided by the robot, was vital in obtaining closed-loop control. Thus, muscle contraction was transmitted to the amplifier through sEMG, activating the mechanism of the robot. Consequentially, sensorial feedback was provided to the patient. Results: Statistically significant improvement of upper limb function was observed in: Fugl-Meyer (p = 0.38) and Box and Block (p = 0.27). The Nine Hole Peg Test did not show statistically significant changes in motor skills of the hand. However, the functional improvement was observed at the level of 6% in the Fugl-Meyer, 15% in the Box and Block, and 2% in the Nine Hole Peg test. Conclusions: Results showed improvement in hand grasp and overall function of the upper limb. Due to sEMG, it was possible to implement robot therapy in the treatment of patients with severe hand impairment.

Pattern Classification of Instantaneous Cognitive Task-load Through GMM Clustering, Laplacian Eigenmap, and Ensemble SVMs

The identification of the temporal variations in human operator cognitive task-load (CTL) is crucial for preventing possible accidents in human-machine collaborative systems. Recent literature has shown that the change of discrete CTL level during human-machine system operations can be objectively recognized using neurophysiological data and supervised learning technique. The objective of this work is to design subject-specific multi-class CTL classifier to reveal the complex unknown relationship between the operator's task performance and neurophysiological features by combining target class labeling, physiological feature reduction and selection, and ensemble classification techniques. The psychophysiological data acquisition experiments were performed under multiple human-machine process control tasks. Four or five target classes of CTL were determined by using a Gaussian mixture model and three human performance variables. By using Laplacian eigenmap, a few salient EEG features were extracted, and heart rates were used as the input features of the CTL classifier. Then, multiple support vector machines were aggregated via majority voting to create an ensemble classifier for recognizing the CTL classes. Finally, the obtained CTL classification results were compared with those of several existing methods. The results showed that the proposed methods are capable of deriving a reasonable number of target classes and low-dimensional optimal EEG features for individual human operator subjects.

Real-time biofeedback device for gait rehabilitation of post-stroke patients

In this work, we develop a device, called 'Walk-Even', that can provide real-time feedback to correct gait asymmetry commonly exhibited in post-stroke survivors and persons with certain neurological disorders. The device computes gait parameters, including gait time, swing time, and stance time of each leg, to detect gait asymmetry and provide corresponding real-time biofeedback by means of auditory and electrotactile stimulation to actively correct the user's gait. The system consists of customized force-sensor-embedded insoles adjustable to fit any shoe size, electrotactile and auditory feedback circuits, microcontroller, and wireless XBee transceivers. The device also offers data saving capability. To validate its accuracy and reliability, we compared the gait measurements from our device with a commercial gait and balance assessment device, Zeno Walkway. The results show good correlation and agreement in a validity study with six healthy subjects and reliability study with seventeen healthy subjects. In addition, preliminary testing on six post-stroke patients after an 8-week training shows that the Walk-Even device helps to improve gait symmetry, foot pressure and forefoot loading of the affected side. Thus, initial testing indicates that the device is accurate in measuring the gait parameters and effective in improving gait symmetry using real-time feedback. The device is portable and low cost and has the potential for use in a non-clinical setting for patients that can walk independently without assistance. A more extensive testing with stroke patients is still ongoing.

Improving Challenge/Skill Ratio in a Multimodal Interface by Simultaneously Adapting Game Difficulty and Haptic Assistance through Psychophysiological and Performance Feedback

In order to harmonize robotic devices with human beings, the robots should be able to perceive important psychosomatic impact triggered by emotional states such as frustration or boredom. This paper presents a new type of biocooperative control architecture, which acts toward improving the challenge/skill relation perceived by the user when interacting with a robotic multimodal interface in a cooperative scenario. In the first part of the paper, open-loop experiments revealed which physiological signals were optimal for inclusion in the feedback loop. These were heart rate, skin conductance level, and skin conductance response frequency. In the second part of the paper, the proposed controller, consisting of a biocooperative architecture with two degrees of freedom, simultaneously modulating game difficulty and haptic assistance through performance and psychophysiological feedback, is presented. With this setup, the perceived challenge can be modulated by means of the game difficulty and the perceived skill by means of the haptic assistance. A new metric (FlowIndex) is proposed to numerically quantify and visualize the challenge/skill relation. The results are contrasted with comparable previously published work and show that the new method afforded a higher FlowIndex (i.e., a superior challenge/skill relation) and an improved balance between augmented performance and user satisfaction (higher level of valence, i.e., a more enjoyable and satisfactory experience).

Cognitive Load Measurement in a Virtual Reality-based Driving System for Autism Intervention

Autism Spectrum Disorder (ASD) is a highly prevalent neurodevelopmental disorder with enormous individual and social cost. In this paper, a novel virtual reality (VR)-based driving system was introduced to teach driving skills to adolescents with ASD. This driving system is capable of gathering eye gaze, electroencephalography, and peripheral physiology data in addition to driving performance data. The objective of this paper is to fuse multimodal information to measure cognitive load during driving such that driving tasks can be individualized for optimal skill learning. Individualization of ASD intervention is an important criterion due to the spectrum nature of the disorder. Twenty adolescents with ASD participated in our study and the data collected were used for systematic feature extraction and classification of cognitive loads based on five well-known machine learning methods. Subsequently, three information fusion schemes-feature level fusion, decision level fusion and hybrid level fusion-were explored. Results indicate that multimodal information fusion can be used to measure cognitive load with high accuracy. Such a mechanism is essential since it will allow individualization of driving skill training based on cognitive load, which will facilitate acceptance of this driving system for clinical use and eventual commercialization.

A Step towards EEG-based brain computer interface for autism intervention

Autism Spectrum Disorder (ASD) is a prevalent and costly neurodevelopmental disorder. Individuals with ASD often have deficits in social communication skills as well as adaptive behavior skills related to daily activities. We have recently designed a novel virtual reality (VR) based driving simulator for driving skill training for individuals with ASD. In this paper, we explored the feasibility of detecting engagement level, emotional states, and mental workload during VR-based driving using EEG as a first step towards a potential EEG-based Brain Computer Interface (BCI) for assisting autism intervention. We used spectral features of EEG signals from a 14-channel EEG neuroheadset, together with therapist ratings of behavioral engagement, enjoyment, frustration, boredom, and difficulty to train a group of classification models. Seven classification methods were applied and compared including Bayes network, naïve Bayes, Support Vector Machine (SVM), multilayer perceptron, K-nearest neighbors (KNN), random forest, and J48. The classification results were promising, with over 80% accuracy in classifying engagement and mental workload, and over 75% accuracy in classifying emotional states. Such results may lead to an adaptive closed-loop VR-based skill training system for use in autism intervention.

Reinforcement learning for adaptive threshold control of restorative brain-computer interfaces: a Bayesian simulation

Restorative brain-computer interfaces (BCI) are increasingly used to provide feedback of neuronal states in a bid to normalize pathological brain activity and achieve behavioral gains. However, patients and healthy subjects alike often show a large variability, or even inability, of brain self-regulation for BCI control, known as BCI illiteracy. Although current co-adaptive algorithms are powerful for assistive BCIs, their inherent class switching clashes with the operant conditioning goal of restorative BCIs. Moreover, due to the treatment rationale, the classifier of restorative BCIs usually has a constrained feature space, thus limiting the possibility of classifier adaptation. In this context, we applied a Bayesian model of neurofeedback and reinforcement learning for different threshold selection strategies to study the impact of threshold adaptation of a linear classifier on optimizing restorative BCIs. For each feedback iteration, we first determined the thresholds that result in minimal action entropy and maximal instructional efficiency. We then used the resulting vector for the simulation of continuous threshold adaptation. We could thus show that threshold adaptation can improve reinforcement learning, particularly in cases of BCI illiteracy. Finally, on the basis of information-theory, we provided an explanation for the achieved benefits of adaptive threshold setting.

Psychophysiological response to cognitive workload during symmetrical, asymmetrical and dual-task walking

Walking with a lower limb prosthesis comes at a high cognitive workload for amputees, possibly affecting their mobility, safety and independency. A biocooperative prosthesis which is able to reduce the cognitive workload of walking could offer a solution. Therefore, we wanted to investigate whether different levels of cognitive workload can be assessed during symmetrical, asymmetrical and dual-task walking and to identify which parameters are the most sensitive. Twenty-four healthy subjects participated in this study. Cognitive workload was assessed through psychophysiological responses, physical and cognitive performance and subjective ratings. The results showed that breathing frequency and heart rate significantly increased, and heart rate variability significantly decreased with increasing cognitive workload during walking (p<.05). Performance measures (e.g., cadence) only changed under high cognitive workload. As a result, psychophysiological measures are the most sensitive to identify changes in cognitive workload during walking. These parameters reflect the cognitive effort necessary to maintain performance during complex walking and can easily be assessed regardless of the task. This makes them excellent candidates to feed to the control loop of a biocooperative prosthesis in order to detect the cognitive workload. This information can then be used to adapt the robotic assistance to the patient's cognitive abilities.

Using "human state aware" robots to enhance physical human-robot interaction in a cooperative scenario

Human motor performance, speed and variability are highly susceptible to emotional states. This paper reviews the impact of the emotions on the motor control performance, and studies the possibility of improving the perceived skill/challenge relation on a multimodal neural rehabilitation scenario, by means of a biocybernetic controller that modulates the assistance provided by a haptic controlled robot in reaction to undesirable physical and mental states. Results from psychophysiological, performance and self assessment data for closed loop experiments in contrast with their open loop counterparts, suggest that the proposed method had a positive impact on the overall challenge/skill relation leading to an enhanced physical human-robot interaction experience.

Motor execution detection based on autonomic nervous system responses

Triggered assistance has been shown to be a successful robotic strategy for provoking motor plasticity, probably because it requires neurologic patients' active participation to initiate a movement involving their impaired limb. Triggered assistance, however, requires sufficient residual motor control to activate the trigger and, thus, is not applicable to individuals with severe neurologic injuries. In these situations, brain and body-computer interfaces have emerged as promising solutions to control robotic devices. In this paper, we investigate the feasibility of a body-machine interface to detect motion execution only monitoring the autonomic nervous system (ANS) response. Four physiological signals were measured (blood pressure, breathing rate, skin conductance response and heart rate) during an isometric pinching task and used to train a classifier based on hidden Markov models. We performed an experiment with six healthy subjects to test the effectiveness of the classifier to detect rest and active pinching periods. The results showed that the movement execution can be accurately classified based only on peripheral autonomic signals, with an accuracy level of 84.5%, sensitivity of 83.8% and specificity of 85.2%. These results are encouraging to perform further research on the use of the ANS response in body-machine interfaces.

A reconfigurable, tendon-based haptic interface for research into human-environment interactions

Human reaction to external stimuli can be investigated in a comprehensive way by using a versatile virtual-reality setup involving multiple display technologies. It is apparent that versatility remains a main challenge when human reactions are examined through the use of haptic interfaces as the interfaces must be able to cope with the entire range of diverse movements and forces/torques a human subject produces. To address the versatility challenge, we have developed a large-scale reconfigurable tendon-based haptic interface which can be adapted to a large variety of task dynamics and is integrated into a Cave Automatic Virtual Environment (CAVE). To prove the versatility of the haptic interface, two tasks, incorporating once the force and once the velocity extrema of a human subject's extremities, were implemented: a simulator with 3-DOF highly dynamic force feedback and a 3-DOF setup optimized to perform dynamic movements. In addition, a 6-DOF platform capable of lifting a human subject off the ground was realized. For these three applications, a position controller was implemented, adapted to each task, and tested. In the controller tests with highly different, task-specific trajectories, the three robot configurations fulfilled the demands on the application-specific accuracy which illustrates and confirms the versatility of the developed haptic interface.