Real-time vision, tactile cues, and visual form agnosia: removing haptic feedback from a "natural" grasping task induces pantomime-like grasps

Coming to grips with reality: Real grasps, but not pantomimed grasps, resist a simultaneous tilt illusion

Investigations of grasping real, 3D objects subjected to illusory effects from a pictorial background often choose in-flight grasp aperture as the primary variable to test the hypothesis that the visuomotor system resists the illusion. Here we test an equally important feature of grasps that has received less attention: in-flight grasp orientation. The current study tested a variant of the simultaneous tilt illusion using a mirror-apparatus to manipulate the availability of haptic feedback. Participants performed grasps with haptic feedback (real grasps) and without it (pantomime grasps), reaching for the reflection of a real, 3D bar atop a background grating that induced a 1.1° bias in the perceived orientation of the bar in a separate sample of participants. Analysis of the hand's in-flight grasp orientation at early, late, and end stages of the reach showed that at no point were the real grasps biased by the illusion. In contrast, pantomimed grasps were affected by the illusion at the late and end stages of the reach. At each stage, the effect on the real grasps was significantly weaker than the effect of the illusion as measured by the mean point of subjective equality (PSE) in a two-alternative forced-choice task. In contrast, the effect on the pantomime grasps was statistically indistinguishable from the mean PSE at all three stages of the reach. These findings reinforce the idea that in-flight grasp orientation, like grasp aperture to pictorial illusions of target size, is refractory to pictorial backgrounds that bias perceived orientation.

Movement kinematic and postural control differences when performing a visuomotor skill in real and virtual environments

Immersive technologies, like virtual and mixed reality, pose a novel challenge for our sensorimotor systems as they deliver simulated sensory inputs that may not match those of the natural environment. These include reduced fields of view, missing or inaccurate haptic information, and distortions of 3D space; differences that may impact the control of motor actions. For instance, reach-to-grasp movements without end-point haptic feedback are characterised by slower and more exaggerated movements. A general uncertainty about sensory input may also induce a more conscious form of movement control. We tested whether a more complex skill like golf putting was also characterized by more consciously controlled movement. In a repeated-measures design, kinematics of the putter swing and postural control were compared between (i) real-world putting, (ii) VR putting, and (iii) VR putting with haptic feedback from a real ball (i.e., mixed reality). Differences in putter swing were observed both between the real world and VR, and between VR conditions with and without haptic information. Further, clear differences in postural control emerged between real and virtual putting, with both VR conditions characterised by larger postural movements, which were more regular and less complex, suggesting a more conscious form of balance control. Conversely, participants actually reported less conscious awareness of their movements in VR. These findings highlight how fundamental movement differences may exist between virtual and natural environments, which may pose challenges for transfer of learning within applications to motor rehabilitation and sport.

Different strategies in pointing tasks and their impact on clinical bedside tests of spatial orientation

Deficits in spatial memory, orientation, and navigation are often early or neglected signs of degenerative and vestibular neurological disorders. A simple and reliable bedside test of these functions would be extremely relevant for diagnostic routine. Pointing at targets in the 3D environment is a basic well-trained common sensorimotor ability that provides a suitable measure. We here describe a smartphone-based pointing device using the built-in inertial sensors for analysis of pointing performance in azimuth and polar spatial coordinates. Interpretation of the vectors measured in this way is not trivial, since the individuals tested may use at least two different strategies: first, they may perform the task in an egocentric eye-based reference system by aligning the fingertip with the target retinotopically or second, by aligning the stretched arm and the index finger with the visual line of sight in allocentric world-based coordinates similar to using a rifle. The two strategies result in considerable differences of target coordinates. A pilot test with a further developed design of the device and an app for a standardized bedside utilization in five healthy volunteers revealed an overall mean deviation of less than 5° between the measured and the true coordinates. Future investigations of neurological patients comparing their performance before and after changes in body position (chair rotation) may allow differentiation of distinct orientational deficits in peripheral (vestibulopathy) or central (hippocampal or cortical) disorders.

People Are Less Susceptible to Illusion When They Use Their Hands to Communicate Rather Than Estimate

When we use our hands to estimate the length of a stick in the Müller-Lyer illusion, we are highly susceptible to the illusion. But when we prepare to act on sticks under the same conditions, we are significantly less susceptible. Here, we asked whether people are susceptible to illusion when they use their hands not to act on objects but to describe them in spontaneous co-speech gestures or conventional sign languages of the deaf. Thirty-two English speakers and 13 American Sign Language signers used their hands to act on, estimate the length of, and describe sticks eliciting the Müller-Lyer illusion. For both gesture and sign, the magnitude of illusion in the description task was smaller than the magnitude of illusion in the estimation task and not different from the magnitude of illusion in the action task. The mechanisms responsible for producing gesture in speech and sign thus appear to operate not on percepts involved in estimation but on percepts derived from the way we act on objects.

Grasping of Real-World Objects Is Not Biased by Ensemble Perception

The visual system is known to extract summary representations of visually similar objects which bias the perception of individual objects toward the ensemble average. Although vision plays a large role in guiding action, less is known about whether ensemble representation is informative for action. Motor behavior is tuned to the veridical dimensions of objects and generally considered resistant to perceptual biases. However, when the relevant grasp dimension is not available or is unconstrained, ensemble perception may be informative to behavior by providing gist information about surrounding objects. In the present study, we examined if summary representations of a surrounding ensemble display influenced grip aperture and orientation when participants reached-to-grasp a central circular target which had an explicit size but importantly no explicit orientation that the visuomotor system could selectively attend to. Maximum grip aperture and grip orientation were not biased by ensemble statistics during grasping, although participants were able to perceive and provide manual estimations of the average size and orientation of the ensemble display. Support vector machine classification of ensemble statistics achieved above-chance classification accuracy when trained on kinematic and electromyography data of the perceptual but not grasping conditions, supporting our univariate findings. These results suggest that even along unconstrained grasping dimensions, visually-guided behaviors toward real-world objects are not biased by ensemble processing.

The Role of Haptic Expectations in Reaching to Grasp: From Pantomime to Natural Grasps and Back Again

When we reach to pick up an object, our actions are effortlessly informed by the object's spatial information, the position of our limbs, stored knowledge of the object's material properties, and what we want to do with the object. A substantial body of evidence suggests that grasps are under the control of "automatic, unconscious" sensorimotor modules housed in the "dorsal stream" of the posterior parietal cortex. Visual online feedback has a strong effect on the hand's in-flight grasp aperture. Previous work of ours exploited this effect to show that grasps are refractory to cued expectations for visual feedback. Nonetheless, when we reach out to pretend to grasp an object (pantomime grasp), our actions are performed with greater cognitive effort and they engage structures outside of the dorsal stream, including the ventral stream. Here we ask whether our previous finding would extend to cued expectations for haptic feedback. Our method involved a mirror apparatus that allowed participants to see a "virtual" target cylinder as a reflection in the mirror at the start of all trials. On "haptic feedback" trials, participants reached behind the mirror to grasp a size-matched cylinder, spatially coincident with the virtual one. On "no-haptic feedback" trials, participants reached behind the mirror and grasped into "thin air" because no cylinder was present. To manipulate haptic expectation, we organized the haptic conditions into blocked, alternating, and randomized schedules with and without verbal cues about the availability of haptic feedback. Replicating earlier work, we found the strongest haptic effects with the blocked schedules and the weakest effects in the randomized uncued schedule. Crucially, the haptic effects in the cued randomized schedule was intermediate. An analysis of the influence of the upcoming and immediately preceding haptic feedback condition in the cued and uncued random schedules showed that cuing the upcoming haptic condition shifted the haptic influence on grip aperture from the immediately preceding trial to the upcoming trial. These findings indicate that, unlike cues to the availability of visual feedback, participants take advantage of cues to the availability of haptic feedback, flexibly engaging pantomime, and natural modes of grasping to optimize the movement.

A summary statistical representation influences perceptions but not visually or memory-guided grasping

A statistical summary representation (SSR) is a phenomenon wherein a target property (e.g., size) is encoded based on the average of the stimulus-set to which it belongs. An SSR has been demonstrated in obligatory judgment tasks; however, to our knowledge no work has examined whether it influences grasps to 3D targets. Here, participants completed a method of adjustment task, and visually and memory-guided grasps in conditions wherein differently sized 3D targets (widths: 20, 30 and 40 mm; height and depth = 10 mm) were presented with equal frequency (i.e., control) and when the smallest (i.e., 20-mm: small-target) and largest (i.e., 40-mm: large-target) targets were presented five times as often as the other targets in the stimulus-set. In the method of adjustment task, responses for the small- and large-target weighting conditions were smaller and larger than the control condition, respectively. In other words, an SSR biased perceptions in the direction of the most frequently presented target in the stimulus-set - a result consistent with the view that perceptions are supported by relative visual information laid down by the ventral visual pathway. In contrast, grip apertures were refractory to target-weighting and was a finding independent of the presence (i.e., visually guided) or absence (i.e., memory-guided) of visual feedback. Furthermore, two one-sided tests showed that peak grip apertures for the different target weighting conditions were within an equivalence boundary. Accordingly, an SSR does not influence 3D grasps and is a finding adding to a growing literature reporting that actions are supported by the absolute visuomotor networks of the dorsal visual pathway.

Adding Haptic Feedback to Virtual Environments With a Cable-Driven Robot Improves Upper Limb Spatio-Temporal Parameters During a Manual Handling Task

Physical interactions within virtual environments are often limited to visual information within a restricted workspace. A new system exploiting a cable-driven parallel robot to combine visual and haptic information related to environmental physical constraints (e.g. shelving, object weight) was developed. The aim of this study was to evaluate the impact on user movement patterns of adding haptic feedback in a virtual environment with this robot. Twelve healthy participants executed a manual handling task under three conditions: 1) in a virtual environment with haptic feedback; 2) in a virtual environment without haptic feedback; 3) in a real physical environment. Temporal parameters (movement time, peak velocity, movement smoothness, time to maximum flexion, time to peak wrist velocity) and spatial parameters of movement (maximum trunk flexion, range of motion of the trunk, length of the trajectory, index of curvature and maximum clearance from the shelf) were analysed during the reaching, lowering and lifting phases. Our results suggest that adding haptic feedback improves spatial parameters of movement to better respect the environmental constraints. However, the visual information presented in the virtual environment through the head mounted display appears to have an impact on temporal parameters of movement leading to greater movement time. Taken together, our results suggest that a cable-driven robot can be a promising device to provide a more ecological context during complex tasks in virtual reality.

A Framework for the Testing and Validation of Simulated Environments in Experimentation and Training

New computer technologies, like virtual reality (VR), have created opportunities to study human behavior and train skills in novel ways. VR holds significant promise for maximizing the efficiency and effectiveness of skill learning in a variety of settings (e.g., sport, medicine, safety-critical industries) through immersive learning and augmentation of existing training methods. In many cases the adoption of VR for training has, however, preceded rigorous testing and validation of the simulation tool. In order for VR to be implemented successfully for both training and psychological experimentation it is necessary to first establish whether the simulation captures fundamental features of the real task and environment, and elicits realistic behaviors. Unfortunately evaluation of VR environments too often confuses presentation and function, and relies on superficial visual features that are not the key determinants of successful training outcomes. Therefore evidence-based methods of establishing the fidelity and validity of VR environments are required. To this end, we outline a taxonomy of the subtypes of fidelity and validity, and propose a variety of practical methods for testing and validating VR training simulations. Ultimately, a successful VR environment is one that enables transfer of learning to the real-world. We propose that key elements of psychological, affective and ergonomic fidelity, are the real determinants of successful transfer. By adopting an evidence-based approach to VR simulation design and testing it is possible to develop valid environments that allow the potential of VR training to be maximized.

Virtually the same? How impaired sensory information in virtual reality may disrupt vision for action

Virtual reality (VR) is a promising tool for expanding the possibilities of psychological experimentation and implementing immersive training applications. Despite a recent surge in interest, there remains an inadequate understanding of how VR impacts basic cognitive processes. Due to the artificial presentation of egocentric distance cues in virtual environments, a number of cues to depth in the optic array are impaired or placed in conflict with each other. Moreover, realistic haptic information is all but absent from current VR systems. The resulting conflicts could impact not only the execution of motor skills in VR but also raise deeper concerns about basic visual processing, and the extent to which virtual objects elicit neural and behavioural responses representative of real objects. In this brief review, we outline how the novel perceptual environment of VR may affect vision for action, by shifting users away from a dorsal mode of control. Fewer binocular cues to depth, conflicting depth information and limited haptic feedback may all impair the specialised, efficient, online control of action characteristic of the dorsal stream. A shift from dorsal to ventral control of action may create a fundamental disparity between virtual and real-world skills that has important consequences for how we understand perception and action in the virtual world.

When perception intrudes on 2D grasping: evidence from Garner interference

When participants reach out to pick up a real 3-D object, their grip aperture reflects the size of the object well before contact is made. At the same time, the classical psychophysical laws and principles of relative size and shape that govern visual perception do not appear to intrude into the control of such movements, which are instead tuned only to the relevant dimension for grasping. In contrast, accumulating evidence suggests that grasps directed at flat 2D objects are not immune to perceptual effects. Thus, in 2D but not 3D grasping, the aperture of the fingers has been shown to be affected by relative and contextual information about the size and shape of the target object. A notable example of this dissociation comes from studies of Garner interference, which signals holistic processing of shape. Previous research has shown that 3D grasping shows no evidence for Garner interference but 2D grasping does (Freud & Ganel, 2015). In a recent study published in this journal (Löhr-Limpens et al., 2019), participants were presented with 2D objects in a Garner paradigm. The pattern of results closely replicated the previously published results with 2D grasping. Unfortunately, the authors, who appear to be unaware the potential differences between 2D and 3D grasping, used their findings to draw an overgeneralized and unwarranted conclusion about the relation between 3D grasping and perception. In this short methodological commentary, we discuss current literature on aperture shaping during 2D grasping and suggest that researchers should play close attention to the nature of the target stimuli they use before drawing conclusions about visual processing for perception and action.

Obeying the law: speed-precision tradeoffs and the adherence to Weber's law in 2D grasping

Visually guided actions toward two-dimensional (2D) and three-dimensional (3D) objects show different patterns of adherence to Weber's law. In 3D grasping, Just Noticeable Differences (JNDs) do not scale with object size, violating Weber's law. Conversely, JNDs in 2D grasping increase with size, showing a pattern of scaler variability between aperture and JND, as predicted by Weber's law. In the current study, we tested whether such scaler variability in 2D grasping reflects genuine adherence to Weber's law. Alternatively, it could be potentially accounted for by a speed-precision tradeoff effect due to an increase in aperture velocity with size. In two experiments, we modified the relation between aperture velocity and size in 2D grasping and tested whether movement trajectories still adhere to Weber's law. In Experiment 1, we aimed to equate aperture velocities between different-sized objects by pre-adjusting the initial finger aperture to match the target's size. In Experiment 2, we reversed the relation between size and velocity by asking participants to hold their fingers wide open prior to grasp, resulting in faster velocities for smaller rather than for larger objects. The results of the two experiments showed that although aperture velocities did not increase with size, adherence to Weber's law was still maintained. These results indicate that the adherence to Weber's law during 2D grasping cannot be accounted for by a speed-precision tradeoff effect, but rather represents genuine reliance on relative, perceptually based computations in visuomotor interactions with 2D objects.

Computer vision technology-based face mirroring system providing mirror therapy for Bell's palsy patients

Purpose: Mirror therapy (MT) is an effective adjunct treatment for Bell's palsy (BP); however, a bifold mirror-based apparatus hindered the effectiveness. Besides, few studies have reported the related factors of facial embodiment. The aim of this study was to evaluate the feasibility of a novel face mirroring System (FMS) and the effects on facial embodiment in BP patients, comparing with conventional mirror book (MB) therapy.Method: This was a within-subject design trial. Thirty-six BP patients were recruited and received investigations on user experience and perception of facial embodiment after each facial task (reset, facial expression, and enunciation) when using both FMS and MB separately and randomly.Results: Data of questionnaires showed FMS had a better user experience and perception of facial embodiment comparing with MB. Patients agreed more strongly on the statements of facial embodiment for facial expression and enunciation tasks when using both apparatuses, comparing with rest.Conclusions: The FMS is a feasible and optimal setup to provide MT for BP patients. Moreover, speech paired motor training is a superior strategy for facilitating facial embodiment.Implications for RehabilitationThe Face Mirroring System is a feasible and optimal apparatus for mirror therapy in Bell's palsy patients.Perception of facial embodiment can be increased via combining multiple sensory feedbacks.Speech paired motor tasks have considerable potential to facilitate facial embodiment.

Chips in the sunshine: color constancy with real versus simulated Munsell chips under illuminants adjacent to the daylight locus

Accurate color judgments rely on a powerful cognitive component. Here we compare the performance of color constancy under real and simulated conditions. Shifts in the u^'v^' color plane induced by illuminant A (2750 K) and illuminant S (>20,000  K) were measured using asymmetric color matching. A general linear model was used to predict performance from the following dependent variables: chroma ("4" and "6"), illuminant ("A" and "S"), presentation mode ("Real" and "Monitor"), and hue zone ("blue," "green," "yellow," "red," and "purple"). There was a strong overall effect [F(7,264)=78.65, p<0.001]. Post hoc analysis showed that performance was substantially superior under real [chromatic constancy index (cCI)=0.76] compared with simulated cCI=0.55) conditions.

Tactile-Based Pantomime Grasping: Knowledge of Results is Not Enough to Support an Absolute Calibration

Tactile-based pantomime-grasping requires that a performer use their right hand to 'grasp' a target previously held in the palm of their opposite hand - a task examining how mechanoreceptive (i.e., tactile) feedback informs the motor system about an object property (i.e., size). Here, we contrasted pantomime-grasps performed with (H+) and without (H-) haptic feedback (i.e., thumb and forefinger position information derived from the grasping hand touching the object) with a condition providing visual KR (VKR) related to absolute target object size. Just-noticeable-difference (JND) scores were computed to determine whether responses adhered to - or violated - Weber's law. JNDs for H+ trials violated the law, whereas H- and VKR trials adhered to the law. Accordingly, results demonstrate that haptic feedback - and not KR - supports an absolute tactile-haptic calibration.

Mental imagery of gravitational motion

There is considerable evidence that gravitational acceleration is taken into account in the interaction with falling targets through an internal model of Earth gravity. Here we asked whether this internal model is accessed also when target motion is imagined rather than real. In the main experiments, naïve participants grasped an imaginary ball, threw it against the ceiling, and caught it on rebound. In different blocks of trials, they had to imagine that the ball moved under terrestrial gravity (1g condition) or under microgravity (0g) as during a space flight. We measured the speed and timing of the throwing and catching actions, and plotted ball flight duration versus throwing speed. Best-fitting duration-speed curves estimate the laws of ball motion implicit in the participant's performance. Surprisingly, we found duration-speed curves compatible with 0g for both the imaginary 0g condition and the imaginary 1g condition, despite the familiarity with Earth gravity effects and the added realism of performing the throwing and catching actions. In a control experiment, naïve participants were asked to throw the imaginary ball vertically upwards at different heights, without hitting the ceiling, and to catch it on its way down. All participants overestimated ball flight durations relative to the durations predicted by the effects of Earth gravity. Overall, the results indicate that mental imagery of motion does not have access to the internal model of Earth gravity, but resorts to a simulation of visual motion. Because visual processing of accelerating/decelerating motion is poor, visual imagery of motion at constant speed or slowly varying speed appears to be the preferred mode to perform the tasks.

Weber's law in 2D and 3D grasping

Visually guided grasping movements directed to real, 3D objects are characterized by a distinguishable trajectory pattern that evades the influence of Weber's law, a basic principle of perception. Conversely, grasping trajectories directed to 2D line drawings of objects adhere to Weber's law. It can be argued, therefore, that during 2D grasping, the visuomotor system fails at operating in analytic mode and is intruded by irrelevant perceptual information. Here, we explored the visual and tactile cues that enable such analytic processing during grasping. In Experiment 1, we compared grasping directed to 3D objects with grasping directed to 2D object photos. Grasping directed to photos adhered to Weber's law, suggesting that richness in visual detail does not contribute to analytic processing. In Experiment 2, we tested whether the visual presentation of 3D objects could support analytic processing even when only partial object-specific tactile information is provided. Surprisingly, grasping could be performed in an analytic fashion, violating Weber's law. In Experiment 3, participants were denied of any haptic feedback at the end of the movement and grasping trajectories again showed adherence to Weber's law. Taken together, the findings suggest that the presentation of real objects combined with indirect haptic information at the end of the movement is sufficient to allow analytic processing during grasp.

Grasping occluded targets: investigating the influence of target visibility, allocentric cue presence, and direction of motion on gaze and grasp accuracy

Participants executed right-handed reach-to-grasp movements toward horizontally translating targets. Visual feedback of the target when reaching, as well as the presence of additional cues placed above and below the target's path, was manipulated. Comparison of average fixations at reach onset and at the time of the grasp suggested that participants accurately extrapolated the occluded target's motion prior to reach onset, but not after the reach had been initiated, resulting in inaccurate grasp placements. Final gaze and grasp positions were more accurate when reaching for leftward moving targets, suggesting individuals use different grasp strategies when reaching for targets traveling away from the reaching hand. Additional cue presence appeared to impair participants' ability to extrapolate the disappeared target's motion, and caused grasps for occluded targets to be less accurate. Novel information is provided about the eye-hand strategies used when reaching for moving targets in unpredictable visual conditions.

Nothing magical: pantomimed grasping is controlled by the ventral system

In a recent amendment to the two-visual-system model, it has been proposed that actions must result in tactile contact with the goal object for the dorsal system to become engaged (Whitwell et al., Neuropsychologia 55:41-50, 2014). The present study tested this addition by assessing the use of allocentric information in normal and pantomime actions. To this end, magicians, and participants who were inexperienced in performing pantomime actions made normal and pantomime grasps toward objects embedded in the Müller-Lyer illusion. During pantomime grasping, a grasp was made next to an object that was in full view (i.e., a displaced pantomime grasping task). The results showed that pantomime grasps took longer, were slower, and had smaller hand apertures than normal grasping. Most importantly, hand apertures were affected by the illusion during pantomime grasping but not in normal grasping, indicating that displaced pantomime grasping is based on allocentric information. This was true for participants without experience in performing pantomime grasps as well as for magicians with experience in pantomiming. The finding that the illusory bias is limited to pantomime grasping and persists with experience supports the conjecture that the normal engagement of the dorsal system's contribution requires tactile contact with a goal object. If no tactile contact is made, then movement control shifts toward the ventral system.

Are We Real When We Fake? Attunement to Object Weight in Natural and Pantomimed Grasping Movements

Behavioral and neuropsychological studies suggest that real actions and pantomimed actions tap, at least in part, different neural systems. Inspired by studies showing weight-attunement in real grasps, here we asked whether (and to what extent) kinematics of pantomimed reach-to-grasp movement can reveal the weight of the pretended target. To address this question, we instructed participants (n = 15) either to grasp or pretend to grasp toward two differently weighted objects, i.e., a light object and heavy object. Using linear discriminant analysis, we then proceeded to classify the weight of the target - either real or pretended - on the basis of the recorded movement patterns. Classification analysis revealed that pantomimed reach-to-grasp movements retained information about object weight, although to a lesser extent than real grasp movements. These results are discussed in relation to the mechanisms underlying the control of real and pantomimed grasping movements.