Discrete Versus Continuous Mapping of Facial Electromyography for Human-Machine Interface Control: Performance and Training Effects

Human-machine interface for two-dimensional steering control with the auricular muscles

Human-machine interfaces (HMIs) can be used to decode a user's motor intention to control an external device. People that suffer from motor disabilities, such as spinal cord injury, can benefit from the uses of these interfaces. While many solutions can be found in this direction, there is still room for improvement both from a decoding, hardware, and subject-motor learning perspective. Here we show, in a series of experiments with non-disabled participants, a novel decoding and training paradigm allowing naïve participants to use their auricular muscles (AM) to control two degrees of freedom with a virtual cursor. AMs are particularly interesting because they are vestigial muscles and are often preserved after neurological diseases. Our method relies on the use of surface electromyographic records and the use of contraction levels of both AMs to modulate the velocity and direction of a cursor in a two-dimensional paradigm. We used a locking mechanism to fix the current position of each axis separately to enable the user to stop the cursor at a certain location. A five-session training procedure (20-30 min per session) with a 2D center-out task was performed by five volunteers. All participants increased their success rate (Initial: 52.78 ± 5.56%; Final: 72.22 ± 6.67%; median ± median absolute deviation) and their trajectory performances throughout the training. We implemented a dual task with visual distractors to assess the mental challenge of controlling while executing another task; our results suggest that the participants could perform the task in cognitively demanding conditions (success rate of 66.67 ± 5.56%). Finally, using the Nasa Task Load Index questionnaire, we found that participants reported lower mental demand and effort in the last two sessions. To summarize, all subjects could learn to control the movement of a cursor with two degrees of freedom using their AM, with a low impact on the cognitive load. Our study is a first step in developing AM-based decoders for HMIs for people with motor disabilities, such as spinal cord injury.

Ability-Based Methods for Personalized Keyboard Generation

This study introduces an ability-based method for personalized keyboard generation, wherein an individual's own movement and human-computer interaction data are used to automatically compute a personalized virtual keyboard layout. Our approach integrates a multidirectional point-select task to characterize cursor control over time, distance, and direction. The characterization is automatically employed to develop a computationally efficient keyboard layout that prioritizes each user's movement abilities through capturing directional constraints and preferences. We evaluated our approach in a study involving 16 participants using inertial sensing and facial electromyography as an access method, resulting in significantly increased communication rates using the personalized keyboard (52.0 bits/min) when compared to a generically optimized keyboard (47.9 bits/min). Our results demonstrate the ability to effectively characterize an individual's movement abilities to design a personalized keyboard for improved communication. This work underscores the importance of integrating a user's motor abilities when designing virtual interfaces.

Face-Computer Interface (FCI): Intent Recognition Based on Facial Electromyography (fEMG) and Online Human-Computer Interface With Audiovisual Feedback

Patients who have lost limb control ability, such as upper limb amputation and high paraplegia, are usually unable to take care of themselves. Establishing a natural, stable, and comfortable human-computer interface (HCI) for controlling rehabilitation assistance robots and other controllable equipments will solve a lot of their troubles. In this study, a complete limbs-free face-computer interface (FCI) framework based on facial electromyography (fEMG) including offline analysis and online control of mechanical equipments was proposed. Six facial movements related to eyebrows, eyes, and mouth were used in this FCI. In the offline stage, 12 models, eight types of features, and three different feature combination methods for model inputing were studied and compared in detail. In the online stage, four well-designed sessions were introduced to control a robotic arm to complete drinking water task in three ways (by touch screen, by fEMG with and without audio feedback) for verification and performance comparison of proposed FCI framework. Three features and one model with an average offline recognition accuracy of 95.3%, a maximum of 98.8%, and a minimum of 91.4% were selected for use in online scenarios. In contrast, the way with audio feedback performed better than that without audio feedback. All subjects completed the drinking task in a few minutes with FCI. The average and smallest time difference between touch screen and fEMG under audio feedback were only 1.24 and 0.37 min, respectively.

Wearable sensors: At the frontier of personalised health monitoring, smart prosthetics and assistive technologies

Wearable sensors have evolved from body-worn fitness tracking devices to multifunctional, highly integrated, compact, and versatile sensors, which can be mounted onto the desired locations of our clothes or body to continuously monitor our body signals, and better interact and communicate with our surrounding environment or equipment. Here, we discuss the latest advances in textile-based and skin-like wearable sensors with a focus on three areas, including (i) personalised health monitoring to facilitate recording physiological signals, body motions, and analysis of body fluids, (ii) smart gloves and prosthetics to realise the sensation of touch and pain, and (iii) assistive technologies to enable disabled people to operate the surrounding motorised equipment using their active organs. We also discuss areas for future research in this emerging field.

Cursor Click Modality in an Accelerometer-Based Computer Access Device

The purpose of this study is to investigate the effects of different cursor click modalities in an alternative computer access device using accelerometry from head tilt to control cursor movement. Eighteen healthy adults performed a target acquisition task using the device with five different cursor click modalities, while maintaining cursor movement control via accelerometry. Three dwell-based click modalities with dwell times of 0.5 s, 1.0 s, and 1.5 s were tested. Two surface electromyography-based click modalities - with the sensor placed next to the eye for a blink and above the eyebrow for a brow raise - were tested. Performance was evaluated using metrics of target selection accuracy, path efficiency, target selection time, and user effort. Surface electromyography-based click modalities were as fast as the shortest dwell time and as accurate as the longest dwell time, and also minimized user effort. Three of the four performance metrics were not affected by sensor location. Future studies will investigate if these results are similar in individuals with neuromuscular disorders.

High-Density Surface Electromyogram-based Biometrics for Personal Identification

Surface electromyogram (sEMG) has been widely applied in neurorehabilitation techniques such as human-machine interface (HMI). The individual difference of sEMG characteristics has long been a challenge for multi-user HMI. However, the individually unique sEMG property indicates its high potential as a biometrics modality. In this work, we propose a novel application of high-density sEMG (HD-sEMG) for personal identification. HD-sEMG can decode the high-resolution spatial patterns of muscle activations, besides the widely studied temporal features, thus providing more sufficient information. We acquired 64-channel HD-sEMG signals on the dorsum of the right hand from 22 subjects during finger muscle isometric contractions. We achieved an accuracy of 99.5% to recognize the identity of each subject, demonstrating the excellent performance of HD-sEMG for personal identification. To the best of our knowledge, this is the first study to employ HD-sEMG for personal identification.Clinical relevance-Our work has proved the huge individual difference of HD-sEMG, which may result from the individually unique bioelectrophysiological activity of human body, deriving from both neural and biomechanical factors. The investigation of subject-specific HD-sEMG pattern may contribute to a better design of subject-specific clinical rehabilitation robots and a deeper understanding of human movement mechanism.

Integrated Head-Tilt and Electromyographic Cursor Control

This study evaluated the performance of two alternate computer access methods that could be used for two-dimensional cursor control. The first method, ACC/sEMG, integrates head acceleration and facial surface electromyography. The second method, Camera Mouse, is a free-to-use, computer vision-based access method. Twenty-four healthy adults performed a target acquisition task using each computer access method across two lighting conditions and three computer orientations. Performance in the task was evaluated using metrics of target selection accuracy, movement time, and path efficiency. Using ACC/sEMG resulted in better mean path efficiency and target selection accuracy, whereas using Camera Mouse resulted in faster target selection. Moreover, performance in the task when using Camera Mouse depended on lighting conditions in the room. The findings of this study show that the ACC/sEMG system is an effective computer access method across different lighting conditions and computer orientations. However, there is a tradeoff between speed and accuracy: ACC/sEMG system provided higher target selection accuracy compared to Camera Mouse, while the latter provided faster target selection. Future development should focus on evaluating performance of each method in populations with limited motor abilities.

Optimized and Predictive Phonemic Interfaces for Augmentative and Alternative Communication

Purpose We empirically assessed the results of computational optimization and prediction in communication interfaces that were designed to allow individuals with severe motor speech disorders to select phonemes and generate speech output. Method Interface layouts were either random or optimized, in which phoneme targets that were likely to be selected together were located in proximity. Target sizes were either static or predictive, such that likely targets were dynamically enlarged following each selection. Communication interfaces were evaluated by 36 users without motor impairments using an alternate access method. Each user was assigned to 1 of 4 interfaces varying in layout and whether prediction was implemented (random/static, random/predictive, optimized/static, optimized/predictive) and participated in 12 sessions over a 3-week period. Six participants with severe motor impairments used both the optimized/static and optimized/predictive interfaces in 1-2 sessions. Results In individuals without motor impairments, prediction provided significantly faster communication rates during training (Sessions 1-9), as users were learning the interface target locations and the novel access method. After training, optimization acted to significantly increase communication rates. The optimization likely became relevant only after training when participants knew the target locations and moved directly to the targets. Participants with motor impairments could use the interfaces with alternate access methods and generally rated the interface with prediction as preferred. Conclusions Optimization and prediction led to increases in communication rates in users without motor impairments. Predictive interfaces were preferred by users with motor impairments. Future research is needed to translate these results into clinical practice. Supplemental Material https://doi.org/10.23641/asha.8636948.

Hands-Free Human-Computer Interface Based on Facial Myoelectric Pattern Recognition

Patients with no or limited hand function usually have difficulty in using conventional input devices such as a mouse or a touch screen. Having the ability of manipulating electronic devices can give patients full access to the digital world, thereby increasing their independence and confidence, and enriching their lives. In this study, a hands-free human-computer interface was developed in order to help patients manipulate computers using facial movements. Five facial movement patterns were detected by four electromyography (EMG) sensors, and classified using myoelectric pattern recognition algorithms. Facial movement patterns were mapped to cursor actions including movements in different directions and click. A typing task and a drawing task were designed in order to assess the interaction performance of the interface in daily use. Ten able-bodied subjects participated in the experiment. In the typing task, the median path efficiency was 80.4%, and the median input rate was 5.9 letters per minute. In the drawing task, the median time to accomplish was 239.9 s. Moreover, all the subjects achieved high classification accuracy (median: 98.0%). The interface driven by facial EMG achieved high performance, and will be assessed on patients with limited hand functions in the future.

Hand Movement Classification Using Burg Reflection Coefficients

Classification of electromyographic signals has a wide range of applications, from clinical diagnosis of different muscular diseases to biomedical engineering, where their use as input for the control of prosthetic devices has become a hot topic of research. The challenge of classifying these signals relies on the accuracy of the proposed algorithm and the possibility of its implementation in hardware. This paper considers the problem of electromyography signal classification, solved with the proposed signal processing and feature extraction stages, with the focus lying on the signal model and time domain characteristics for better classification accuracy. The proposal considers a simple preprocessing technique that produces signals suitable for feature extraction and the Burg reflection coefficients to form learning and classification patterns. These coefficients yield a competitive classification rate compared to the time domain features used. Sometimes, the feature extraction from electromyographic signals has shown that the procedure can omit less useful traits for machine learning models. Using feature selection algorithms provides a higher classification performance with as few traits as possible. The algorithms achieved a high classification rate up to 100% with low pattern dimensionality, with other kinds of uncorrelated attributes for hand movement identification.

Prediction of Optimal Facial Electromyographic Sensor Configurations for Human-Machine Interface Control

Surface electromyography (sEMG) is a promising computer access method for individuals with motor impairments. However, optimal sensor placement is a tedious task requiring trial-and-error by an expert, particularly when recording from facial musculature likely to be spared in individuals with neurological impairments. We sought to reduce the sEMG sensor configuration complexity by using quantitative signal features extracted from a short calibration task to predict human-machine interface (HMI) performance. A cursor control system allowed individuals to activate specific sEMG-targeted muscles to control an onscreen cursor and navigate a target selection task. The task was repeated for a range of sensor configurations to elicit a range of signal qualities. Signal features were extracted from the calibration of each configuration and examined via a principle component factor analysis in order to predict the HMI performance during subsequent tasks. Feature components most influenced by the energy and the complexity of the EMG signal and muscle activity between the sensors were significantly predictive of the HMI performance. However, configuration order had a greater effect on performance than the configurations, suggesting that non-experts can place sEMG sensors in the vicinity of usable muscle sites for computer access and healthy individuals will learn to efficiently control the HMI system.

A Novel Feature Optimization for Wearable Human-Computer Interfaces Using Surface Electromyography Sensors

The novel human-computer interface (HCI) using bioelectrical signals as input is a valuable tool to improve the lives of people with disabilities. In this paper, surface electromyography (sEMG) signals induced by four classes of wrist movements were acquired from four sites on the lower arm with our designed system. Forty-two features were extracted from the time, frequency and time-frequency domains. Optimal channels were determined from single-channel classification performance rank. The optimal-feature selection was according to a modified entropy criteria (EC) and Fisher discrimination (FD) criteria. The feature selection results were evaluated by four different classifiers, and compared with other conventional feature subsets. In online tests, the wearable system acquired real-time sEMG signals. The selected features and trained classifier model were used to control a telecar through four different paradigms in a designed environment with simple obstacles. Performance was evaluated based on travel time (TT) and recognition rate (RR). The results of hardware evaluation verified the feasibility of our acquisition systems, and ensured signal quality. Single-channel analysis results indicated that the channel located on the extensor carpi ulnaris (ECU) performed best with mean classification accuracy of 97.45% for all movement’s pairs. Channels placed on ECU and the extensor carpi radialis (ECR) were selected according to the accuracy rank. Experimental results showed that the proposed FD method was better than other feature selection methods and single-type features. The combination of FD and random forest (RF) performed best in offline analysis, with 96.77% multi-class RR. Online results illustrated that the state-machine paradigm with a 125 ms window had the highest maneuverability and was closest to real-life control. Subjects could accomplish online sessions by three sEMG-based paradigms, with average times of 46.02, 49.06 and 48.08 s, respectively. These experiments validate the feasibility of proposed real-time wearable HCI system and algorithms, providing a potential assistive device interface for persons with disabilities.

Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.

Effects of muscle fatigue on the usability of a myoelectric human-computer interface

Electromyography-based human-computer interface development is an active field of research. However, knowledge on the effects of muscle fatigue for specific devices is limited. We have developed a novel myoelectric human-computer interface in which subjects continuously navigate a cursor to targets by manipulating a single surface electromyography (sEMG) signal. Two-dimensional control is achieved through simultaneous adjustments of power in two frequency bands through a series of dynamic low-level muscle contractions. Here, we investigate the potential effects of muscle fatigue during the use of our interface. In the first session, eight subjects completed 300 cursor-to-target trials without breaks; four using a wrist muscle and four using a head muscle. The wrist subjects returned for a second session in which a static fatiguing exercise took place at regular intervals in-between cursor-to-target trials. In the first session we observed no declines in performance as a function of use, even after the long period of use. In the second session, we observed clear changes in cursor trajectories, paired with a target-specific decrease in hit rates.

Surface electromyographic control of a novel phonemic interface for speech synthesis

Many individuals with minimal movement capabilities use AAC to communicate. These individuals require both an interface with which to construct a message (e.g., a grid of letters) and an input modality with which to select targets. This study evaluated the interaction of two such systems: (a) an input modality using surface electromyography (sEMG) of spared facial musculature, and (b) an onscreen interface from which users select phonemic targets. These systems were evaluated in two experiments: (a) participants without motor impairments used the systems during a series of eight training sessions, and (b) one individual who uses AAC used the systems for two sessions. Both the phonemic interface and the electromyographic cursor show promise for future AAC applications.

Learning to modulate the partial powers of a single sEMG power spectrum through a novel human-computer interface

New human-computer interfaces that use bioelectrical signals as input are allowing study of the flexibility of the human neuromuscular system. We have developed a myoelectric human-computer interface which enables users to navigate a cursor to targets through manipulations of partial powers within a single surface electromyography (sEMG) signal. Users obtain two-dimensional control through simultaneous adjustments of powers in two frequency bands within the sEMG spectrum, creating power profiles corresponding to cursor positions. It is unlikely that these types of bioelectrical manipulations are required during routine muscle contractions. Here, we formally establish the neuromuscular ability to voluntarily modulate single-site sEMG power profiles in a group of naïve subjects under restricted and controlled conditions using a wrist muscle. All subjects used the same pre-selected frequency bands for control and underwent the same training, allowing a description of the average learning progress throughout eight sessions. We show that subjects steadily increased target hit rates from 48% to 71% and exhibited greater control of the cursor's trajectories following practice. Our results point towards an adaptable neuromuscular skill, which may allow humans to utilize single muscle sites as limited general-purpose signal generators. Ultimately, the goal is to translate this neuromuscular ability to practical interfaces for the disabled by using a spared muscle to control external machines.