Quantitative Eye Gaze and Movement Differences in Visuomotor Adaptations to Varying Task Demands Among Upper-Extremity Prosthesis Users

Comparing eye-hand coordination between controller-mediated virtual reality, and a real-world object interaction task

Virtual reality (VR) technology has advanced significantly in recent years, with many potential applications. However, it is unclear how well VR simulations mimic real-world experiences, particularly in terms of eye-hand coordination. This study compares eye-hand coordination from a previously validated real-world object interaction task to the same task re-created in controller-mediated VR. We recorded eye and body movements and segmented participants' gaze data using the movement data. In the real-world condition, participants wore a head-mounted eye tracker and motion capture markers and moved a pasta box into and out of a set of shelves. In the VR condition, participants wore a VR headset and moved a virtual box using handheld controllers. Unsurprisingly, VR participants took longer to complete the task. Before picking up or dropping off the box, participants in the real world visually fixated the box about half a second before their hand arrived at the area of action. This 500-ms minimum fixation time before the hand arrived was preserved in VR. Real-world participants disengaged their eyes from the box almost immediately after their hand initiated or terminated the interaction, but VR participants stayed fixated on the box for much longer after it was picked up or dropped off. We speculate that the limited haptic feedback during object interactions in VR forces users to maintain visual fixation on objects longer than in the real world, altering eye-hand coordination. These findings suggest that current VR technology does not replicate real-world experience in terms of eye-hand coordination.

Generating accurate 3D gaze vectors using synchronized eye tracking and motion capture

Assessing gaze behavior during real-world tasks is difficult; dynamic bodies moving through dynamic worlds make gaze analysis difficult. Current approaches involve laborious coding of pupil positions. In settings where motion capture and mobile eye tracking are used concurrently in naturalistic tasks, it is critical that data collection be simple, efficient, and systematic. One solution is to combine eye tracking with motion capture to generate 3D gaze vectors. When combined with tracked or known object locations, 3D gaze vector generation can be automated. Here we use combined eye and motion capture and explore how linear regression models generate accurate 3D gaze vectors. We compare spatial accuracy of models derived from four short calibration routines across three pupil data inputs: the efficacy of calibration routines was assessed, a validation task requiring short fixations on task-relevant locations, and a naturalistic object interaction task to bridge the gap between laboratory and "in the wild" studies. Further, we generated and compared models using spherical and Cartesian coordinate systems and monocular (left or right) or binocular data. All calibration routines performed similarly, with the best performance (i.e., sub-centimeter errors) coming from the naturalistic task trials when the participant is looking at an object in front of them. We found that spherical coordinate systems generate the most accurate gaze vectors with no differences in accuracy when using monocular or binocular data. Overall, we recommend 1-min calibration routines using binocular pupil data combined with a spherical world coordinate system to produce the highest-quality gaze vectors.

Sensory Restoration for Improved Motor Control of Prostheses

Somatosensory neuroprostheses are devices with the potential to restore the senses of touch and movement from prosthetic limbs for people with limb amputation or paralysis. By electrically stimulating the peripheral or central nervous system, these devices evoke sensations that appear to emanate from the missing or insensate limb, and when paired with sensors on the prosthesis, they can improve the functionality and embodiment of the prosthesis. There have been major advances in the design of these systems over the past decade, although several important steps remain before they can achieve widespread clinical adoption outside the lab setting. Here, we provide a brief overview of somatosensory neuroprostheses and explores these hurdles and potential next steps towards clinical translation.

Nonlinear Mapping From EMG to Prosthesis Closing Velocity Improves Force Control With EMG Biofeedback

When using EMG biofeedback to control the grasping force of a myoelectric prosthesis, subjects need to activate their muscles and maintain the myoelectric signal within an appropriate interval. However, their performance decreases for higher forces, because the myoelectric signal is more variable for stronger contractions. Therefore, the present study proposes to implement EMG biofeedback using nonlinear mapping, in which EMG intervals of increasing size are mapped to equal-sized intervals of the prosthesis velocity. To validate this approach, 20 non-disabled subjects performed force-matching tasks using Michelangelo prosthesis with and without EMG biofeedback with linear and nonlinear mapping. Additionally, four transradial amputees performed a functional task in the same feedback and mapping conditions. The success rate in producing desired force was significantly higher with feedback (65.4±15.9%) compared to no feedback (46.2±14.9%) as well as when using nonlinear (62.4±16.8%) versus linear mapping (49.2±17.2%). Overall, in non-disabled subjects, the highest success rate was obtained when EMG biofeedback was combined with nonlinear mapping (72%), and the opposite for linear mapping with no feedback (39.6%). The same trend was registered also in four amputee subjects. Therefore, EMG biofeedback improved prosthesis force control, especially when combined with nonlinear mapping, which showed to be an effective approach to counteract increasing variability of myoelectric signal for stronger contractions.

A scoping review of eye tracking metrics used to assess visuomotor behaviours of upper limb prosthesis users

Advanced upper limb prostheses aim to restore coordinated hand and arm function. However, this objective can be difficult to quantify as coordinated movements require an intact visuomotor system. Eye tracking has recently been applied to study the visuomotor behaviours of upper limb prosthesis users by enabling the calculation of eye movement metrics. This scoping review aims to characterize the visuomotor behaviours of upper limb prosthesis users as described by eye tracking metrics, to summarize the eye tracking metrics used to describe prosthetic behaviour, and to identify gaps in the literature and potential areas for future research. A review of the literature was performed to identify articles that reported eye tracking metrics to evaluate the visual behaviours of individuals using an upper limb prosthesis. Data on the level of amputation, type of prosthetic device, type of eye tracker, primary eye metrics, secondary outcome metrics, experimental task, aims, and key findings were extracted. Seventeen studies were included in this scoping review. A consistently reported finding is that prosthesis users have a characteristic visuomotor behaviour that differs from that of individuals with intact arm function. Visual attention has been reported to be directed more towards the hand and less towards the target during object manipulation tasks. A gaze switching strategy and delay to disengage gaze from the current target has also been reported. Differences in the type of prosthetic device and experimental task have revealed some distinct gaze behaviours. Control factors have been shown to be related to gaze behaviour, while sensory feedback and training interventions have been demonstrated to reduce the visual attention associated with prosthesis use. Eye tracking metrics have also been used to assess the cognitive load and sense of agency of prosthesis users. Overall, there is evidence that eye tracking is an effective tool to quantitatively assess the visuomotor behaviour of prosthesis users and the recorded eye metrics are sensitive to change in response to various factors. Additional studies are needed to validate the eye metrics used to assess cognitive load and sense of agency in upper limb prosthesis users.

Limb loading enhances skill transfer between augmented and physical reality tasks during limb loss rehabilitation

BACKGROUND

Virtual and augmented reality (AR) have become popular modalities for training myoelectric prosthesis control with upper-limb amputees. While some systems have shown moderate success, it is unclear how well the complex motor skills learned in an AR simulation transfer to completing the same tasks in physical reality. Limb loading is a possible dimension of motor skill execution that is absent in current AR solutions that may help to increase skill transfer between the virtual and physical domains.

METHODS

We implemented an immersive AR environment where individuals could operate a myoelectric virtual prosthesis to accomplish a variety of object relocation manipulations. Intact limb participants were separated into three groups, the load control (CG_LD; [Formula: see text]), the AR control (CG_AR; [Formula: see text]), and the experimental group (EG; [Formula: see text]). Both the CG_AR and EG completed a 5-session prosthesis training protocol in AR while the CG_LD performed simple muscle training. The EG attempted manipulations in AR while undergoing limb loading. The CG_AR attempted the same manipulations without loading. All participants performed the same manipulations in physical reality while operating a real prosthesis pre- and post-training. The main outcome measure was the change in the number of manipulations completed during the physical reality assessments (i.e. completion rate). Secondary outcomes included movement kinematics and visuomotor behavior.

RESULTS

The EG experienced a greater increase in completion rate post-training than both the CG_AR and CG_LD. This performance increase was accompanied by a shorter motor learning phase, the EG's performance saturating in less sessions of AR training than the CG_AR.

CONCLUSION

The results demonstrated that limb loading plays an important role in transferring complex motor skills learned in virtual spaces to their physical reality analogs. While participants who did not receive limb loading were able to receive some functional benefit from AR training, participants who received the loading experienced a greater positive change in motor performance with their performance saturating in fewer training sessions.

Factors influencing perceived function in the upper limb prosthesis user population

BACKGROUND

Patient-reported outcomes (PROs) can be used to evaluate perceived capacity of an individual in executing tasks in a natural environment with their prosthetic device. According to the World Health Organization International Classification of Health, Functioning, and Disability (ICF) models, there may be specific factors of a person, factors of assistive prosthetic technology, or factors related to the health condition or body function that affect their functioning and disability. However, an understanding of factors affecting an upper limb prosthesis user's perception of their ability to execute tasks in a natural environment is not well established.

OBJECTIVE

To use the ICF model to identify which health condition-related, body function, environmental, and personal factors influence activity as measured by perceived function in the upper limb prosthesis user population.

DESIGN

Quantitative clinical descriptive study.

SETTING

Clinical offices within outpatient private practice (removed for blinding).

PARTICIPANTS

A sample of 101 participants with upper limb amputation who use a prosthetic device and were undergoing a prosthesis fitting process.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

PROs on pain with/without a prosthesis, satisfaction, and perceived function derived from the Comprehensive Arm Prosthesis and Rehabilitation Outcomes Questionnaire.

RESULTS

Model coefficients indicate that with a unit increase in satisfaction (p < .001) and pain (p = .031) scores (with higher pain scores signifying less pain), the mean of perceived function increases by 0.66 and 0.47 units, respectively. Conversely, for individuals with elbow disarticulation, transhumeral, shoulder disarticulation, and interscapulothoracic amputations, the mean of perceived function decreases by 22.02 units (p = .006).

CONCLUSIONS

Based on our sample, perceived function is significantly associated with satisfaction, pain, and amputation level. These findings could potentially help to inform initial clinical approach and targeted outcomes for patients based on these factors.

Communicative capital: a key resource for human-machine shared agency and collaborative capacity

In this work, we present a perspective on the role machine intelligence can play in supporting human abilities. In particular, we consider research in rehabilitation technologies such as prosthetic devices, as this domain requires tight coupling between human and machine. Taking an agent-based view of such devices, we propose that human-machine collaborations have a capacity to perform tasks which is a result of the combined agency of the human and the machine. We introduce communicative capital as a resource developed by a human and a machine working together in ongoing interactions. Development of this resource enables the partnership to eventually perform tasks at a capacity greater than either individual could achieve alone. We then examine the benefits and challenges of increasing the agency of prostheses by surveying literature which demonstrates that building communicative resources enables more complex, task-directed interactions. The viewpoint developed in this article extends current thinking on how best to support the functional use of increasingly complex prostheses, and establishes insight toward creating more fruitful interactions between humans and supportive, assistive, and augmentative technologies.

Integrating computer vision to prosthetic hand control with sEMG: Preliminary results in grasp classification

The myoelectric prosthesis is a promising tool to restore the hand abilities of amputees, but the classification accuracy of surface electromyography (sEMG) is not high enough for real-time application. Researchers proposed integrating sEMG signals with another feature that is not affected by amputation. The strong coordination between vision and hand manipulation makes us consider including visual information in prosthetic hand control. In this study, we identified a sweet period during the early reaching phase in which the vision data could yield a higher accuracy in classifying the grasp patterns. Moreover, the visual classification results from the sweet period could be naturally integrated with sEMG data collected during the grasp phase. After the integration, the accuracy of grasp classification increased from 85.5% (only sEMG) to 90.06% (integrated). Knowledge gained from this study encourages us to further explore the methods for incorporating computer vision into myoelectric data to enhance the movement control of prosthetic hands.

Spatiotemporal Coupling of Hand and Eye Movements When Using a Myoelectric Prosthetic Hand

Upper limb prosthesis users have disruptions in hand-eye coordination, with increased fixations towards the hand and less visual allocation for feedforward planning. The purpose of this study was to explore whether improved motor planning, as reflected by eye gaze behaviour, was associated with more efficient hand movement patterns. Able-bodied participants wore a simulated prosthesis while performing a functional object movement task. Motion and eye tracking data were collected to quantify the eye gaze and hand movement during object interaction. The results of this study demonstrated that the latency of the eye to precede the hand at pick-up was correlated with measures of hand function, including hand variability, movement units, and grasp time, but not reach time. During transport and release, longer latency to disengage gaze from the grasped object and look ahead towards the target was correlated to hand kinematics of hand variability, distance travelled, and transport time. In addition, the latency of the eye to disengage the drop-off location was correlated to release time. Together these may point to control issues with opening and closing the prosthetic hand. Overall, increased feedforward fixations towards the target and reduced feedback fixations towards the hand were related to improved measures of hand function. Hence, coordination between eye and hand movements when using a myoelectric prosthesis may prove to be a useful metric to assess motor planning.

A comparison of compensatory movements between body-powered and myoelectric prosthesis users during activities of daily living

BACKGROUND

People with upper limb absence use compensatory movements to accommodate lack of motion in the prosthetic hand. The purpose of this study was to determine if the type of prosthesis used (i.e. body-powered or myoelectric) affects compensatory movements during activities of daily living.

METHODS

Twelve transradial body-powered and/or myoelectric prosthesis users performed up to six unimanual and bimanual activities of daily living. Trunk range of motion and peak upper limb angles for each task were compared between prostheses.

FINDINGS

Compensatory movement generally did not differ based on prosthesis type. However, body-powered users had increased trunk lateral lean compared to myoelectric users during a deodorant application task (P = 0.025). Body-powered users also had increased trunk axial rotation (P = 0.048) and decreased shoulder elevation (P = 0.046) when transferring a box between shelves. Compensatory movements were not systematically correlated with duration of prosthesis ownership, socket comfort, or terminal device type.

INTERPRETATION

A prosthesis user's compensatory movements may depend on other factors beyond whether the prosthesis terminal device is actuated through body-powered or myoelectric mechanisms. Further exploration of the factors that influence joint kinematics in prosthesis users may inform future prosthesis prescription practices and help patients become successful users.

Neurorobotic fusion of prosthetic touch, kinesthesia, and movement in bionic upper limbs promotes intrinsic brain behaviors

[Figure: see text].

Myoelectric prosthesis users and non-disabled individuals wearing a simulated prosthesis exhibit similar compensatory movement strategies

Background

Research studies on upper limb prosthesis function often rely on the use of simulated myoelectric prostheses (attached to and operated by individuals with intact limbs), primarily to increase participant sample size. However, it is not known if these devices elicit the same movement strategies as myoelectric prostheses (operated by individuals with amputation). The objective of this study was to address the question of whether non-disabled individuals using simulated prostheses employ the same compensatory movements (measured by hand and upper body kinematics) as individuals who use actual myoelectric prostheses.

Methods

The upper limb movements of two participant groups were investigated: (1) twelve non-disabled individuals wearing a simulated prosthesis, and (2) three individuals with transradial amputation using their custom-fitted myoelectric devices. Motion capture was used for data collection while participants performed a standardized functional task. Performance metrics, hand movements, and upper body angular kinematics were calculated. For each participant group, these measures were compared to those from a normative baseline dataset. Each deviation from normative movement behaviour, by either participant group, indicated that compensatory movements were used during task performance.

Results

Results show that participants using either a simulated or actual myoelectric prosthesis exhibited similar deviations from normative behaviour in phase durations, hand velocities, hand trajectories, number of movement units, grip aperture plateaus, and trunk and shoulder ranges of motion.

Conclusions

This study suggests that the use of a simulated prosthetic device in upper limb research offers a reasonable approximation of compensatory movements employed by a low- to moderately-skilled transradial myoelectric prosthesis user.

What’s limbs got to do with it? Real-world movement correlates with feelings of ownership over virtual arms during object interactions in virtual reality

Humans will initially move awkwardly so that the end-state of their movement is comfortable. But, what is comfortable? We might assume it refers to a particular physical body posture, however, humans have been shown to move a computer cursor on a screen with an out-of-sight hand less efficiently (curved) such that the visual representation appears more efficient (straight). This suggests that movement plans are made in large part to satisfy the demands of their visual appearance, rather than their physical movement properties. So, what determines if a body movement is comfortable—how it feels or how it looks? We translated an object-interaction task from the real-world into immersive virtual reality (IVR) to dissociate a movement from its visual appearance. Participants completed at least 20 trials in two conditions: Controllers—where participants saw a visual representation of the hand-held controllers and Arms—where they saw a set of virtual limbs. We found participants seeing virtual limbs moved in a less biomechanically efficient manner to make the limbs look similar to if they were interacting with a real-world object. These movement changes correlated with an increase in self-reported feelings of ownership over the limbs as compared to the controllers. Overall this suggests we plan our movements to provide optimal visual feedback, even at the cost of being less efficient. Moreover, we speculate that a detailed measurement of how people move in IVR may provide a new tool for assessing their degree of embodiment. There is something about seeing a set of limbs in front of you, doing your actions, that affects your moving, and in essence, your thinking.

Application of machine learning to the identification of joint degrees of freedom involved in abnormal movement during upper limb prosthesis use

To evaluate movement quality of upper limb (UL) prosthesis users, performance-based outcome measures have been developed that examine the normalcy of movement as compared to a person with a sound, intact hand. However, the broad definition of “normal movement” and the subjective nature of scoring can make it difficult to know which areas of the body to evaluate, and the expected magnitude of deviation from normative movement. To provide a more robust approach to characterizing movement differences, the goals of this work are to identify degrees of freedom (DOFs) that will inform abnormal movement for several tasks using unsupervised machine learning (clustering methods) and elucidate the variations in movement approach across two upper-limb prosthesis devices with varying DOFs as compared to healthy controls. 24 participants with no UL disability or impairment were recruited for this study and trained on the use of a body-powered bypass (n = 6) or the DEKA limb bypass (n = 6) prosthetic devices or included as normative controls. 3D motion capture data were collected from all participants as they performed the Jebsen-Taylor Hand Function Test (JHFT) and targeted Box and Blocks Test (tBBT). Range of Motion, peak angle, angular path length, mean angle, peak angular velocity, and number of zero crossings were calculated from joint angle data for the right/left elbows, right/left shoulders, torso, and neck and fed into a K-means clustering algorithm. Results show right shoulder and torso DOFs to be most informative in distinguishing between bypass user and norm group movement. The JHFT page turning task and the seated tBBT elicit movements from bypass users that are most distinctive from the norm group. Results can be used to inform the development of movement quality scoring methodology for UL performance-based outcome measures. Identifying tasks across two different devices with known variations in movement can inform the best tasks to perform in a rehabilitation setting that challenge the prosthesis user’s ability to achieve normative movement.

Addressing unpredictability may be the key to improving performance with current clinically prescribed myoelectric prostheses

The efferent control chain for an upper-limb myoelectric prosthesis can be separated into 3 key areas: signal generation, signal acquisition, and device response. Data were collected from twenty trans-radial myoelectric prosthesis users using their own clinically prescribed devices, to establish the relative impact of these potential control factors on user performance (user functionality and everyday prosthesis usage). By identifying the key factor(s), we can guide future developments to ensure clinical impact. Skill in generating muscle signals was assessed via reaction times and signal tracking. To assess the predictability of signal acquisition, we inspected reaction time spread and undesired hand activations. As a measure of device response, we recorded the electromechanical delay between electrode stimulation and the onset of hand movement. Results suggest abstract measures of skill in controlling muscle signals are poorly correlated with performance. Undesired activations of the hand or incorrect responses were correlated with almost all kinematics and gaze measures suggesting unpredictability is a key factor. Significant correlations were also found between several measures of performance and the electromechanical delay; however, unexpectedly, longer electromechanical delays correlated with better performance. Future research should focus on exploring causes of unpredictability, their relative impacts on performance and interventions to address this.

A Modular Adjustable Transhumeral Prosthetic Socket for Evaluating Myoelectric Control

Novel myoelectric control strategies may yield more robust, capable prostheses which improve quality of life for those affected by upper-limb loss; however, the development and translation of such strategies from an experimental setting towards daily use by persons with limb loss is a slow and costly process. Since prosthesis functionality is highly dependent on the physical interface between the user’s prosthetic socket and residual limb, assessment of such controllers under realistic (noisy) environmental conditions, integrated into prosthetic sockets, and with participants with amputation is essential for obtaining representative results. Unfortunately, this step is particularly difficult as participant- and control strategy-specific prosthetic sockets must be custom-designed and manufactured. There is thus a need for a system to reduce these burdens and facilitate this crucial phase of the development pipeline. This study aims to address this gap through the design and assessment of an inexpensive and easy-to-use 3D-printed Modular-Adjustable transhumeral Prosthetic Socket (MAPS). This 3D-printed, open-source socket was developed in consultation with prosthetists and compared with a participant-specific suction socket in a single-participant case-study. We conducted mechanical and functional assessments to ensure that the developed socket enabled similar performance compared to participant-specific sockets. Both socket systems yielded similar results in mechanical and functional assessments, as well as in self-reported user feedback. The MAPS system shows promise as a research tool which catalyzes the development and deployment of novel myoelectric control strategies by better-enabling comprehensive assessment involving participants with amputations.

Mechanotactile Sensory Feedback Improves Embodiment of a Prosthetic Hand During Active Use

There have been several advancements in the field of myoelectric prostheses to improve dexterity and restore hand grasp patterns for persons with upper limb loss, including robust control strategies, novel sensory feedback, and multifunction prosthetic terminal devices. Although these advancements have shown to improve prosthesis performance, a key element that may further improve acceptance is often overlooked. Embodiment, which encompasses the feeling of owning, controlling and locating the device without the need to constantly look at it, has been shown to be affected by sensory feedback. However, the specific aspects of embodiment that are influenced are not clearly understood, particularly when a prosthesis is actively controlled. In this work, we used a sensorized simulated prosthesis in able-bodied participants to investigate the contribution of sensory feedback, active motor control, and the combination of both to the components of embodiment; using a common methodology in the literature, namely the rubber hand illusion (RHI). Our results indicate that (1) the sensorized simulated prosthesis may be embodied by able-bodied users in a similar fashion as prosthetic devices embodied by persons with upper limb amputation, and (2) mechanotactile sensory feedback might not only be useful for improving certain aspects of embodiment, i.e., ownership and location, but also may have a modulating effect on other aspects, namely sense of agency, when provided asynchronously during active motor control tasks. This work may allow us to further investigate and manipulate factors contributing to the complex phenomenon of embodiment in relation to active motor control of a device, enabling future study of more precise quantitative measures of embodiment that do not rely as much on subjective perception.

On the Visuomotor Behavior of Amputees and Able-Bodied People During Grasping

Visual attention is often predictive for future actions in humans. In manipulation tasks, the eyes tend to fixate an object of interest even before the reach-to-grasp is initiated. Some recent studies have proposed to exploit this anticipatory gaze behavior to improve the control of dexterous upper limb prostheses. This requires a detailed understanding of visuomotor coordination to determine in which temporal window gaze may provide helpful information. In this paper, we verify and quantify the gaze and motor behavior of 14 transradial amputees who were asked to grasp and manipulate common household objects with their missing limb. For comparison, we also include data from 30 able-bodied subjects who executed the same protocol with their right arm. The dataset contains gaze, first person video, angular velocities of the head, and electromyography and accelerometry of the forearm. To analyze the large amount of video, we developed a procedure based on recent deep learning methods to automatically detect and segment all objects of interest. This allowed us to accurately determine the pixel distances between the gaze point, the target object, and the limb in each individual frame. Our analysis shows a clear coordination between the eyes and the limb in the reach-to-grasp phase, confirming that both intact and amputated subjects precede the grasp with their eyes by more than 500 ms. Furthermore, we note that the gaze behavior of amputees was remarkably similar to that of the able-bodied control group, despite their inability to physically manipulate the objects.