Force feedback facilitates multisensory integration during robotic tool use

Effect of force-rate on continuous kinesthetic force discrimination

The effect of force-rate, i.e., rate of change of force stimuli, on continuous kinesthetic force perception has not been investigated and remains an open area for research. Previous studies do not account for the force-rate in the Weber fraction of kinesthetic force discrimination. However, this force-rate agnostic Weber fraction fails to explain continuous kinesthetic force discrimination fully. For example, if the signal changes very slowly, a participant may get accustomed to the change, and hence, a larger, just noticeable difference (JND) is expected. Conversely, for a fast-varying haptic force, a smaller JND is expected. In this work, we aim to explore the relationship between the Weber fraction and the force-rate. For this purpose, we designed an extensive psychophysical experiment where a participant is exposed to a linearly increasing kinesthetic force stimulus and is asked to react to the change. We utilize a machine learning-based approach to study the effect of force-rate on recorded haptic responses (perceived and non-perceived) of 10 participants while varying the force-rate stimuli in the range [1, 5] N/s. We determine the perceptual boundary between the perceived and non-perceived recorded responses using different classifiers based on linear and power functions of force-rate. The result indicates that the Weber fraction decreases significantly as the force-rate increases. The random forest classifier also confirms the significance of the utilized features in both perceptual boundaries. These findings may be useful in many virtual reality applications and telepresence and teleaction systems (TPTA).

Findings Favor Haptics Feedback in Virtual Simulation Surgical Education: An Updated Systematic and Scoping Review

BACKGROUND

Virtual simulations (VSs) enhance clinical competencies and skills. However, a previous systematic review of 9 RCT studies highlighted a paucity of literature on the effects of haptic feedback in surgical VSs. An updated systematic and scoping review was conducted to encompass more studies and a broader range of study methodologies.

METHODS

A systematic literature search was conducted on July 31, 2023, in MEDLINE, Embase, and Cochrane. English language studies comparing haptic vs non-haptic conditions and using VSs were included. Studies were evaluated and reported using PRISMA-ScR guidelines.

RESULTS

Out of 2782 initial studies, 51 were included in the review. Most studies used RCT (21) or crossover (23) methodologies with medical residents, students, and attending physicians. Most used post-intervention metrics, while some used pre- and post-intervention metrics. Overall, 34 performance results from studies favored haptics, 3 favored non-haptics, and the rest showed mixed or equal results.

CONCLUSION

This updated review highlights the diverse application of haptic technology in surgical VSs. Haptics generally enhances performance, complements traditional teaching methods, and offers personalized learning with adequate simulator validation. However, a sparsity of orienting to the simulator, pre-/post-study designs, and small sample sizes poses concerns with the validity of the results. We underscore the urgent need for standardized protocols, large-scale studies, and nuanced understanding of haptic feedback integration. We also accentuate the significance of simulator validation, personalized learning potential, and the need for researcher, educator, and manufacturer collaboration. This review is a guidepost for navigating the complexities and advancements in haptic-enhanced surgical VSs.

Sensory substitution increases robotic surgical performance and sets the ground for a mediating role of the sense of embodiment: a systematic review

Sensory Substitution (SS) allows the elaboration of information via non preferential sensory modalities. This phenomenon occurs in robotic-assisted surgery (RAS), in which haptic feedback is lacking. It has been suggested that SS could sustain surgeons' proficiency by means of visual clues for inferring tactile information, that also promotes the feeling of haptic phantom sensations. A critical role in reaching a good performance in procedural tasks is also sustained by the Sense of Embodiment (SE), that is, the capacity to integrate objects into subjective bodily self-representation. As SE is enhanced by haptic sensations, we hypothesize a role of SS in promoting SE in RAS. Accordingly, the goal of this systematic review is to summarize the evidence pertaining the study of SS in RAS in order to highlight the impact on the performance, and to identify a mediating role of the SE in increasing dexterity in RAS. Eight studies selected from the MEDLINE and Scopus® databases met inclusion criteria for a qualitative synthesis. Results indicated that haptic to other modalities SS enhanced force consistency and accuracy, and decreased surgeon fatigue. Expert surgeons, as compared to novices, showed a better natural SS processing, testified by a proficient performance with and without SS aids. No studies investigated the mediating role of SE. These findings indicate that SS is subjected to learning and memory processes that help surgeons to rapidly derive haptic-correlates from visual clues, which are highly required for a good performance. Also, the higher ability of doing SS and the associated perception of haptic sensations might increase multisensory integration, which might sustain performance.

Impact of supplementary sensory feedback on the control and embodiment in human movement augmentation

In human movement augmentation, the number of controlled degrees of freedom could be enhanced by the simultaneous and independent use of supernumerary robotic limbs (SRL) and natural ones. However, this poses several challenges, that could be mitigated by encoding and relaying the SRL status. Here, we review the impact of supplementary sensory feedback on the control and embodiment of SRLs. We classify the main feedback features and analyse how they improve control performance. We report the feasibility of pushing body representation beyond natural human morphology and suggest that gradual SRL embodiment could make multisensory incongruencies less disruptive. We also highlight shared computational bases between SRL motor control and embodiment and suggest contextualizing them within the same theoretical framework. Finally, we argue that a shift towards long term experimental paradigms is necessary for successfully integrating motor control and embodiment.

Supernumerary robotic limbs are robotic devices providing additional limbs to the user. Mattia Pinardi and colleagues review the impact of supplementary sensory feedback on the control performance and embodiment of supernumerary robotic limbs.

Neural signatures of visuo-motor integration during human-robot interactions

Visuo-motor integration shapes our daily experience and underpins the sense of feeling in control over our actions. The last decade has seen a surge in robotically and virtually mediated interactions, whereby bodily actions ultimately result in an artificial movement. But despite the growing number of applications, the neurophysiological correlates of visuo-motor processing during human-machine interactions under dynamic conditions remain scarce. Here we address this issue by employing a bimanual robotic interface able to track voluntary hands movement, rendered in real-time into the motion of two virtual hands. We experimentally manipulated the visual feedback in the virtual reality with spatial and temporal conflicts and investigated their impact on (1) visuo-motor integration and (2) the subjective experience of being the author of one's action (i.e., sense of agency). Using somatosensory evoked responses measured with electroencephalography, we investigated neural differences occurring when the integration between motor commands and visual feedback is disrupted. Our results show that the right posterior parietal cortex encodes for differences between congruent and spatially-incongruent interactions. The experimental manipulations also induced a decrease in the sense of agency over the robotically-mediated actions. These findings offer solid neurophysiological grounds that can be used in the future to monitor integration mechanisms during movements and ultimately enhance subjective experience during human-machine interactions.

The Remapping of Peripersonal Space in a Real but Not in a Virtual Environment

One of the most surprising features of our brain is the fact that it is extremely plastic. Among the various plastic processes supported by our brain, there is the neural representation of the space surrounding our body, the peripersonal space (PPS). The effects of real-world tool use on the PPS are well known in cognitive neuroscience, but little is still known whether similar mechanisms also govern virtual tool use. To this purpose, the present study investigated the plasticity of the PPS before and after a real (Experiment 1) or virtual motor training with a tool (Experiment 2). The results show the expansion of the PPS only following real-world tool use but not virtual use, highlighting how the two types of training potentially rely on different processes. This study enriches the current state of the art on the plasticity of PPS in real and virtual environments. We discuss our data with respect to the relevance for the development of effective immersive environment for trainings, learning and rehabilitation.

Do Multisensory Stimuli Benefit the Virtual Reality Experience? A Systematic Review

The majority of virtual reality (VR) applications rely on audiovisual stimuli and do not exploit the addition of other sensory cues that could increase the potential of VR. This systematic review surveys the existing literature on multisensory VR and the impact of haptic, olfactory, and taste cues over audiovisual VR. The goal is to identify the extent to which multisensory stimuli affect the VR experience, which stimuli are used in multisensory VR, the type of VR setups used, and the application fields covered. An analysis of the 105 studies that met the eligibility criteria revealed that 84.8 percent of the studies show a positive impact of multisensory VR experiences. Haptics is the most commonly used stimulus in multisensory VR systems (86.6 percent). Non-immersive and immersive VR setups are preferred over semi-immersive setups. Regarding the application fields, a considerable part was adopted by health professionals and science and engineering professionals. We further conclude that smell and taste are still underexplored, and they can bring significant value to VR applications. More research is recommended on how to synthesize and deliver these stimuli, which still require complex and costly apparatus be integrated into the VR experience in a controlled and straightforward manner.

Vision is biased near handheld, but not remotely operated, tools

Previous studies have shown that after actively using a handheld tool for a period of time, participants show visual biases toward stimuli presented near the end of the tool. Research suggests this is driven by an incorporation of the tool into the observer's body schema, extending peripersonal space to surround the tool. This study aims to investigate whether the same visual biases might be seen near remotely operated tools. Participants used tools-a handheld rake (Experiment 1), a remote-controlled drone (Experiment 2), a remote-controlled excavator (Experiment 3), or a handheld excavator (Experiment 4)-to rake sand for several minutes, then performed a target-detection task in which they made speeded responses to targets appearing near and far from the tool. In Experiment 1, participants detected targets appearing near the rake significantly faster than targets appearing far from the rake, replicating previous findings. We failed to find strong evidence of improved target detection near remotely operated tools in Experiments 2 and 3, but found clear evidence of near-tool facilitation in Experiment 4 when participants physically picked up the excavator and used it as a handheld tool. These results suggest that observers may not incorporate remotely operated tools into the body schema in the same manner as handheld tools. We discuss potential mechanisms that may drive these differences in embodiment between handheld and remote-controlled tools.

The Embodiment of Objects: Review, Analysis, and Future Directions

Here we offer a thorough review of the empirical literature on the conditions under which an object, such as a tool or a prosthetic (whether real or virtual), can be experienced as being in some sense a part or extension of one's body. We discuss this literature both from the standpoint of the apparent malleability of our body representations, and also from within the framework of radical embodied cognition, which understands the phenomenon to result not from an alteration to a representation, but rather from the achievement of a certain kind of sensory/motor coupling. We highlight both the tensions between these frameworks, and also areas where they can productively complement one another for future research.

Tool Use Modulates Somatosensory Cortical Processing in Humans

Tool use leads to plastic changes in sensorimotor body representations underlying tactile perception. The neural correlates of this tool-induced plasticity in humans have not been adequately characterized. This study used ERPs to investigate the stage of sensory processing modulated by tool use. Somatosensory evoked potentials, elicited by median nerve stimulation, were recorded before and after two forms of object interaction: tool use and hand use. Compared with baseline, tool use-but not use of the hand alone-modulated the amplitude of the P100. The P100 is a mid-latency component that indexes the construction of multisensory models of the body and has generators in secondary somatosensory and posterior parietal cortices. These results mark one of the first demonstrations of the neural correlates of tool-induced plasticity in humans and suggest that tool use modulates relatively late stages of somatosensory processing outside primary somatosensory cortex. This finding is consistent with what has been observed in tool-trained monkeys and suggests that the mechanisms underlying tool-induced plasticity have been preserved across primate evolution.

Crossmodal congruency effect scores decrease with repeat test exposure

The incorporation of feedback into a person’s body schema is well established. The crossmodal congruency task (CCT) is used to objectively quantify incorporation without being susceptible to experimenter biases. This visual-tactile interference task is used to calculate the crossmodal congruency effect (CCE) score as a difference in response time between incongruent and congruent trials. Here we show that this metric is susceptible to a learning effect that causes attenuation of the CCE score due to repeated task exposure sessions. We demonstrate that this learning effect is persistent, even after a 6 month hiatus in testing. Two mitigation strategies are proposed: 1. Only use CCE scores that are taken after learning has stabilized, or 2. Use a modified CCT protocol that decreases the task exposure time. We show that the modified and shortened CCT protocol, which may be required to meet time or logistical constraints in laboratory or clinical settings, reduced the impact of the learning effect on CCT results. Importantly, the CCE scores from the modified protocol were not significantly more variable than results obtained with the original protocol. This study highlights the importance of considering exposure time to the CCT when designing experiments and suggests two mitigation strategies to improve the utility of this psychophysical assessment.

The recalibration of tactile perception during tool use is body-part specific

Two decades of research have demonstrated that using a tool modulates spatial representations of the body. Whether this embodiment is specific to representations of the tool-using limb or extends to representations of other body parts has received little attention. Several studies of other perceptual phenomena have found that modulations to the primary somatosensory representation of the hand transfers to the face, due in part to their close proximity in primary somatosensory cortex. In the present study, we investigated whether tool-induced recalibration of tactile perception on the hand transfers to the cheek. Participants verbally estimated the distance between two tactile points applied to either their hand or face, before and after using a hand-shaped tool. Tool use recalibrated tactile distance perception on the hand-in line with previous findings-but left perception on the cheek unchanged. This finding provides support for the idea that embodiment is body-part specific. Furthermore, it suggests that tool-induced perceptual recalibration occurs at a level of somatosensory processing, where representations of the hand and face have become functionally disentangled.

The contribution of response conflict, multisensory integration, and body-mediated attention to the crossmodal congruency effect

The crossmodal congruency task is a consolidated paradigm for investigating interactions between vision and touch. In this task, participants judge the elevation of a tactile target stimulus while ignoring a visual distracter stimulus that may occur at a congruent or incongruent elevation, thus engendering a measure of visuo-tactile interference (crossmodal congruency effect, CCE). The CCE reflects perceptual, attentional, and response-related factors, but their respective roles and interactions have not been set out yet. In two experiments, we used the original version of the crossmodal congruency task as well as ad hoc manipulations of the experimental setting and of the participants' posture for characterizing the contributions of multisensory integration, body-mediated attention, and response conflict to the CCE. Results of the two experiments consistently showed that the largest amount of variance in the CCE is explained by the reciprocal elevation of visual and tactile stimuli. This finding is compatible with a major role of response conflict for the CCE. Weaker yet distinguishable contributions come from multisensory integration, observed in the absence of response conflict, and from hand-mediated attentional binding, observed with the modified posture and in the presence of response conflict. Overall, this study informs the long-standing debate about mechanisms underlying the CCE by revealing that the visuo-tactile interference in this task is primarily due to the competition between opposite response tendencies, with an additional contribution of multisensory integration and hand-mediated attentional binding.

In a demanding task, three-handed manipulation is preferred to two-handed manipulation

Equipped with a third hand under their direct control, surgeons may be able to perform certain surgical interventions alone; this would reduce the need for a human assistant and related coordination difficulties. However, does human performance improve with three hands compared to two hands? To evaluate this possibility, we carried out a behavioural study on the performance of naive adults catching objects with three virtual hands controlled by their two hands and right foot. The subjects could successfully control the virtual hands in a few trials. With this control strategy, the workspace of the hands was inversely correlated with the task velocity. The comparison of performance between the three and two hands control revealed no significant difference of success in catching falling objects and in average effort during the tasks. Subjects preferred the three handed control strategy, found it easier, with less physical and mental burden. Although the coordination of the foot with the natural hands increased trial after trial, about two minutes of practice was not sufficient to develop a sense of ownership towards the third arm.

Haptic over visual information in the distribution of visual attention after tool-use in near and far space

Despite attentional prioritization for grasping space near the hands, tool-use appears to transfer attentional bias to the tool's end/functional part. The contributions of haptic and visual inputs to attentional distribution along a tool were investigated as a function of tool-use in near (Experiment 1) and far (Experiment 2) space. Visual attention was assessed with a 50/50, go/no-go, target discrimination task, while a tool was held next to targets appearing near the tool-occupied hand or tool-end. Target response times (RTs) and sensitivity (d-prime) were measured at target locations, before and after functional tool practice for three conditions: (1) open-tool: tool-end visible (visual + haptic inputs), (2) hidden-tool: tool-end visually obscured (haptic input only), and (3) short-tool: stick missing tool's length/end (control condition: hand occupied but no visual/haptic input). In near space, both open- and hidden-tool groups showed a tool-end, attentional bias (faster RTs toward tool-end) before practice; after practice, RTs near the hand improved. In far space, the open-tool group showed no bias before practice; after practice, target RTs near the tool-end improved. However, the hidden-tool group showed a consistent tool-end bias despite practice. Lack of short-tool group results suggested that hidden-tool group results were specific to haptic inputs. In conclusion, (1) allocation of visual attention along a tool due to tool practice differs in near and far space, and (2) visual attention is drawn toward the tool's end even when visually obscured, suggesting haptic input provides sufficient information for directing attention along the tool.

Robot-assisted surgery: an emerging platform for human neuroscience research

Classic studies in human sensorimotor control use simplified tasks to uncover fundamental control strategies employed by the nervous system. Such simple tasks are critical for isolating specific features of motor, sensory, or cognitive processes, and for inferring causality between these features and observed behavioral changes. However, it remains unclear how these theories translate to complex sensorimotor tasks or to natural behaviors. Part of the difficulty in performing such experiments has been the lack of appropriate tools for measuring complex motor skills in real-world contexts. Robot-assisted surgery (RAS) provides an opportunity to overcome these challenges by enabling unobtrusive measurements of user behavior. In addition, a continuum of tasks with varying complexity-from simple tasks such as those in classic studies to highly complex tasks such as a surgical procedure-can be studied using RAS platforms. Finally, RAS includes a diverse participant population of inexperienced users all the way to expert surgeons. In this perspective, we illustrate how the characteristics of RAS systems make them compelling platforms to extend many theories in human neuroscience, as well as, to develop new theories altogether.