Seeing all the obstacles in your way: the effect of visual feedback and visual feedback schedule on obstacle avoidance while reaching

Online updating of obstacle positions when intercepting a virtual target

People rely upon sensory information in the environment to guide their actions. Ongoing goal-directed arm movements are constantly adjusted to the latest estimate of both the target and hand's positions. Does the continuous guidance of ongoing arm movements also consider the latest visual information of the position of obstacles in the surrounding? To find out, we asked participants to slide their finger across a screen to intercept a laterally moving virtual target while moving through a gap that was created by two virtual circular obstacles. At a fixed time during each trial, the target suddenly jumped slightly laterally while still continuing to move. In half the trials, the size of the gap changed at the same moment as the target jumped. As expected, participants adjusted their movements in response to the target jump. Importantly, the magnitude of this response depended on the new size of the gap. If participants were told that the circles were irrelevant, changing the gap between them had no effect on the responses. This shows that obstacles' instantaneous positions can be considered when visually guiding goal-directed movements.

Eye-hand coordination: memory-guided grasping during obstacle avoidance

When reaching to grasp previously seen, now out-of-view objects, we rely on stored perceptual representations to guide our actions, likely encoded by the ventral visual stream. So-called memory-guided actions are numerous in daily life, for instance, as we reach to grasp a coffee cup hidden behind our morning newspaper. Little research has examined obstacle avoidance during memory-guided grasping, though it is possible obstacles with increased perceptual salience will provoke exacerbated avoidance maneuvers, like exaggerated deviations in eye and hand position away from obtrusive obstacles. We examined the obstacle avoidance strategies adopted as subjects reached to grasp a 3D target object under visually-guided (closed loop or open loop with full vision prior to movement onset) and memory-guided (short- or long-delay) conditions. On any given trial, subjects reached between a pair of flanker obstacles to grasp a target object. The positions and widths of the obstacles were manipulated, though their inner edges remained a constant distance apart. While reach and grasp behavior was consistent with the obstacle avoidance literature, in that reach, grasp, and gaze positions were biased away from obstacles most obtrusive to the reaching hand, our results reveal distinctive avoidance approaches undertaken depend on the availability of visual feedback. Contrary to expectation, we found subjects reaching to grasp after a long delay in the absence of visual feedback failed to modify their final fixation and grasp positions to accommodate the different positions of obstacles, demonstrating a more moderate, rather than exaggerative, obstacle avoidance strategy.

Reaching for known unknowns: Rapid reach decisions accurately reflect the future state of dynamic probabilistic information

Everyday tasks such as catching a ball appear effortless, but in fact require complex interactions and tight temporal coordination between the brain's visual and motor systems. What makes such interceptive actions particularly impressive is the capacity of the brain to account for temporal delays in the central nervous system-a limitation that can be mitigated by making predictions about the environment as well as one's own actions. Here, we wanted to assess how well human participants can plan an upcoming movement based on a dynamic, predictable stimulus that is not the target of action. A central stationary or rotating stimulus determined the probability that each of two potential targets would be the eventual target of a rapid reach-to-touch movement. We examined the extent to which reach movement trajectories convey internal predictions about the future state of dynamic probabilistic information conveyed by the rotating stimulus. We show that movement trajectories reflect the target probabilities determined at movement onset, suggesting that humans rapidly and accurately integrate visuospatial predictions and estimates of their own reaction times to effectively guide action.

Blindsight

For over a century, research has demonstrated that damage to primary visual cortex does not eliminate all capacity for visual processing in the brain. From Riddoch's (1917) early demonstration of intact motion processing for blind field stimuli, to the iconic work of Weiskrantz et al. (1974) showing reliable spatial localization, it is clear that secondary visual pathways that bypass V1 carry information to the visual brain that in turn influences behavior. In this chapter, we briefly outline the history and phenomena associated with blindsight, before discussing the nature of the secondary visual pathways that support residual visual processing in the absence of V1. We finish with some speculation as to the functional characteristics of these secondary pathways.

Reaching movements are attracted by stimuli that signal reward

When presented with a set of possible reach targets, the movement trajectory can reveal aspects of the underlying competition for action selection. Current goals and physical salience can affect the trajectory of reaching movements to be attracted towards a distractor. Some studies demonstrated that stimuli associated with reward can also cause an attraction when reaching towards the reward stimulus was previously rewarded and the reward stimulus was physically salient. Here we demonstrate that a non-salient stimulus that signals the availability of reward attracts reaching movements even when moving towards it was never necessary nor rewarded. Moreover, the attraction by reward is particularly evident with short-latency movements. We conclude that neither physical salience nor reinforcing the movement towards a stimulus is necessary for reward to gain priority in the selection for action.

The attractiveness of salient distractors to reaching movements is task dependent

Previous studies in visual attention and oculomotor research showed that a physically salient distractor does not always capture attention or the eyes. Under certain top-down task sets, a salient distractor can be actively suppressed, avoiding capture. Even though previous studies showed that reaching movements are also influenced by salient distractors, it is unclear if and how a mechanism of active suppression of distractors would affect reaching movements. Active suppression might also explain why some studies find reaching movements to curve towards a distractor, while others find reaching movements to curve away. In this study, we varied the top-down task set in two separate experiments by manipulating the certainty about the target location. Participants had to reach for a diamond present among three circles. In Experiments 1 and 3, participants had to search for the reach targets; hence, the target's location certainty was low. In Experiments 2 and 3, the target's location was cued before the reach; hence, the target's location certainty was high. We found that reaches curved towards the physically salient, color singleton, distractor in the search-to-reach task (Experiments 1 and 3), but not in the cued reach task (Experiments 2 and 3). Thus, the saliency of the distractor only attracted reaching movements when the certainty of the target's location was low. Our findings suggest that the attractiveness of physically salient distractors to reaching movements depends on the top-down task set. The results can be explained by the effect of active attentional suppression on the competition between movement plans.

Visual Selection of the Future Reach Path in Obstacle Avoidance

In two EEG experiments, we studied the role of visual attention during the preparation of manual movements around an obstacle. Participants performed rapid hand movements to a goal position avoiding a central obstacle either on the left or right side, depending on the pitch of the acoustical go signal. We used a dot probe paradigm to analyze the deployment of spatial attention in the visual field during the motor preparation. Briefly after the go signal but still before the hand movement actually started, a visual transient was flashed either on the planned pathway of the hand (congruent trials) or on the opposite, movement-irrelevant side (incongruent trials). The P1/N1 components that were evoked by the onset of the dot probe were enhanced in congruent trials where the visual transient was presented on the planned path of the hand. The results indicate that, during movement preparation, attention is allocated selectively to the planned trajectory the hand is going to take around the obstacle.

Saccades and reaches curve away from the other effector’s target in simultaneous eye and hand movements

Simultaneous eye and hand movements are highly coordinated and tightly coupled. This raises the question whether the selection of eye and hand targets relies on a shared attentional mechanism or separate attentional systems. Previous studies have revealed conflicting results by reporting evidence for both a shared as well as separate systems. Movement properties such as movement curvature can provide novel insights into this question as they provide a sensitive measure for attentional allocation during target selection. In the current study, participants performed simultaneous eye and hand movements to the same or different visual target locations. We show that both saccade and reaching movements curve away from the other effector’s target location when they are simultaneously performed to spatially distinct locations. We argue that there is a shared attentional mechanism involved in selecting eye and hand targets that may be found on the level of effector-independent priority maps.

NEW & NOTEWORTHY Movement properties such as movement curvature have been widely neglected as important sources of information in investigating whether the attentional systems underlying target selection for eye and hand movements are separate or shared. We convincingly show that movement curvature is influenced by the other effector’s target location in simultaneous eye and hand movements to spatially distinct locations. Our results provide evidence for shared attentional systems involved in the selection of saccade and reach targets.

Avoiding unseen obstacles: Subcortical vision is not sufficient to maintain normal obstacle avoidance behaviour during reaching

Previous research found that a patient with cortical blindness (homonymous hemianopia) was able to successfully avoid an obstacle placed in his blind field, despite reporting no conscious awareness of it [Striemer, C. L., Chapman, C. S., & Goodale, M. A., 2009, PNAS, 106(37), 15996-16001]. This finding led to the suggestion that dorsal stream areas, that are assumed to mediate obstacle avoidance behaviour, may obtain their visual input primarily from subcortical pathways. Hence, it was suggested that normal obstacle avoidance behaviour can proceed without input from the primary visual cortex. Here we tried to replicate this finding in a group of patients (N = 6) that suffered from highly circumscribed lesions in the occipital lobe (including V1) that spared the subcortical structures that have been associated with action-blindsight. We also tested if obstacle avoidance behaviour differs depending on whether obstacles are placed only in the blind field or in both the blind and intact visual field of the patients simultaneously. As expected, all patients successfully avoided obstacles placed in their intact visual field. However, none of them showed reliable avoidance behaviour - as indicated by adjustments in the hand trajectory in response to obstacle position - for obstacles placed in their blind visual field. The effects were not dependent on whether one or two obstacles were present. These findings suggest that behaviour in complex visuomotor tasks relies on visual input from occipital areas.

The contributions of vision and haptics to reaching and grasping

This review aims to provide a comprehensive outlook on the sensory (visual and haptic) contributions to reaching and grasping. The focus is on studies in developing children, normal, and neuropsychological populations, and in sensory-deprived individuals. Studies have suggested a right-hand/left-hemisphere specialization for visually guided grasping and a left-hand/right-hemisphere specialization for haptically guided object recognition. This poses the interesting possibility that when vision is not available and grasping relies heavily on the haptic system, there is an advantage to use the left hand. We review the evidence for this possibility and dissect the unique contributions of the visual and haptic systems to grasping. We ultimately discuss how the integration of these two sensory modalities shape hand preference.

Use of early phase online vision for grip configuration is modulated according to movement duration in prehension

Our previous study (Hum Mov Sci 25:349-371, 2006) investigated whether and how online vision in the early phase of movement influences the control of reach-to-grasp movements (movement duration: approximately 1000 ms). We used liquid-crystal shutter goggles to manipulate the duration of available online vision during the movement and specified that online vision during the early phase influences grasping movements. The current study examined the effect of online early phase vision on the grip configuration according to the movement duration and compared it between two different movement durations (approximately 500 and 1000 ms). We found that non-availability of early phase online vision affected the grip configuration (i.e., inducing a larger peak grip aperture) even in the shorter movement duration. The influential period for online vision for grasping control shifts to an earlier time when movement time is shorter (i.e., from approximately 214 to 106 ms after movement onset), indicating a flexible mechanism for grip configuration according to the movement duration and the available online vision.

Hitting a target is fundamentally different from avoiding obstacles

To successfully move our hand to a target, it is important not only to consider the target of our movements but also to consider other objects in the environment that may act as obstacles. We previously found that the time needed to respond to a change in position was considerably longer for a displacement of an obstacle than for a displacement of the target (Aivar, Brenner, & Smeets, 2008. Experimental Brain Research 190, 251-264). In that study, the movement constraints imposed by the obstacles differed from those imposed by the target. To examine whether the latency is really different for targets and obstacles, irrespective of any constraints they impose, we modified the design of the previous experiment to make sure that the constraints were matched. In each trial, two aligned 'objects' of the same size were presented at different distances to the left of the initial position of the hand. Each of these objects could either be a target or a gap (opening between two obstacles). Participants were instructed to pass through both objects. All possible combinations of these two objects were tested: gap-target, target-gap, gap-gap, target-target. On some trials one of the objects changed position after movement onset. Participants systematically responded faster to the displacement of a target than to the displacement of a gap at the same location. We conclude that targets are prioritized over obstacles in movement control.

Non-obstructing 3D depth cues influence reach-to-grasp kinematics

It has been demonstrated that both visual feedback and the presence of certain types of non-target objects in the workspace can affect kinematic measures and the trajectory path of the moving hand during reach-to-grasp movements. Yet no study to date has examined the possible effect of providing non-obstructing three-dimensional (3D) depth cues within the workspace and with consistent retinal inputs and whether or not these alter manual prehension movements. Participants performed a series of reach-to-grasp movements in both open- (without visual feedback) and closed-loop (with visual feedback) conditions in the presence of one of three possible 3D depth cues. Here, it is reported that preventing online visual feedback (or not) and the presence of a particular depth cue had a profound effect on kinematic measures for both the reaching and grasping components of manual prehension-despite the fact that the 3D depth cues did not act as a physical obstruction at any point. The depth cues modulated the trajectory of the reaching hand when the target block was located on the left side of the workspace but not on the right. These results are discussed in relation to previous reports and implications for brain-computer interface decoding algorithms are provided.

Outsider interference: no role for motor lateralization in determining the strength of avoidance responses during reaching

When reaches are performed toward target objects, the presence of other non-target objects influences kinematic parameters of the reach. A typical observation has been that non-targets positioned ipsilaterally to the acting limb interfere more with the trajectory of the hand than contralateral non-targets. Here, we investigate whether this effect is mediated by motor lateralization or by the relative positioning of the objects with reference to the acting limb. Participants were asked to perform reaches toward physical target objects with their preferred or non-preferred hands while physical non-targets were present in different possible positions in the workspace. We tested both left-handers and right-handers. Our results show that a participant's handedness does not influence reaching behavior in an obstacle avoidance paradigm. Furthermore, no statistically significant differences between the use of the preferred and non-preferred hand were observed on the kinematic parameters of the reaches. We found evidence that non-targets positioned on the outside of the reaching limb influenced the reaching behavior more strongly than non-targets on the inside. Moreover, the type of movement also appeared to play a role, as reaches that crossed the workspace had a stronger effect on avoidance behavior than reaches that were 'uncrossed.' We interpret these results as support for the hypothesis that the avoidance response is determined by keeping a preferred distance between the acting limb in all stages of its reach toward the target and the non-target position. This process is not biased by hand dominance or the hand preference of the actor.

Gaze-centered spatial updating in delayed reaching even in the presence of landmarks

Previous results suggest that the brain predominantly relies on a constantly updated gaze-centered target representation to guide reach movements when no other visual information is available. In the present study, we investigated whether the addition of reliable visual landmarks influences the use of spatial reference frames for immediate and delayed reaching. Subjects reached immediately or after a delay of 8 or 12s to remembered target locations, either with or without landmarks. After target presentation and before reaching they shifted gaze to one of five different fixation points and held their gaze at this location until the end of the reach. With landmarks present, gaze-dependent reaching errors were smaller and more precise than when reaching without landmarks. Delay influenced neither reaching errors nor variability. These findings suggest that when landmarks are available, the brain seems to still use gaze-dependent representations but combine them with gaze-independent allocentric information to guide immediate or delayed reach movements to visual targets.

Speeded reaching movements around invisible obstacles

We analyze the problem of obstacle avoidance from a Bayesian decision-theoretic perspective using an experimental task in which reaches around a virtual obstacle were made toward targets on an upright monitor. Subjects received monetary rewards for touching the target and incurred losses for accidentally touching the intervening obstacle. The locations of target-obstacle pairs within the workspace were varied from trial to trial. We compared human performance to that of a Bayesian ideal movement planner (who chooses motor strategies maximizing expected gain) using the Dominance Test employed in Hudson et al. (2007). The ideal movement planner suffers from the same sources of noise as the human, but selects movement plans that maximize expected gain in the presence of that noise. We find good agreement between the predictions of the model and actual performance in most but not all experimental conditions.

In everyday, cluttered environments, moving to reach or grasp an object can result in unintended collisions with other objects along the path of movement. Depending on what we run into (a priceless Ming vase, a crotchety colleague) we can suffer serious monetary or social consequences. It makes sense to choose movement trajectories that trade off the value of reaching a goal against the consequences of unintended collisions along the way. In the research described here, subjects made speeded movements to touch targets while avoiding obstacles placed along the natural reach trajectory. There were explicit monetary rewards for hitting the target and explicit monetary costs for accidentally hitting the intervening obstacle. We varied the cost and location of the obstacle across conditions. The task was to earn as large a monetary bonus as possible, which required that reaches curve around obstacles only to the extent justified by the location and cost of the obstacle. We compared human performance in this task to that of a Bayesian movement planner who maximized expected gain on each trial. In most conditions, but not all, movement strategies were close to optimal.

Mental blocks: fMRI reveals top-down modulation of early visual cortex when obstacles interfere with grasp planning

When grasping an object, the fingers, hand and arm rarely collide with other non-target objects in the workspace. Kinematic studies of neurological patients (Schindler et al., 2004) and healthy participants (Chapman and Goodale, 2010a) suggest that the location of potential obstacles and the degree of interference they pose are encoded by the dorsal visual stream during action planning. Here, we used a slow event-related paradigm in functional magnetic resonance imaging (fMRI) to examine the neural encoding of obstacles in normal participants. Fifteen right-handed participants grasped a square target object with a thumb-front or thumb-side wrist-posture with (1) no obstacle present, (2) an obstacle behind the target object (interfering with the thumb-front grasp), or (3) an obstacle beside the target object (interfering with the thumb-side grasp). Within a specified network of areas involved in planning, a group voxelwise analysis revealed that one area in the left posterior intraparietal sulcus (pIPS) and one in early visual cortex were modulated by the degree of obstacle interference, and that this modulation occurred prior to movement execution. Given previous reports of a functional link between IPS and early visual cortex, we suggest that the increasing activity in the IPS with obstacle interference provides the top-down signal to suppress the corresponding obstacle coding in early visual areas, where we observed that activity decreased with interference. This is the first concrete evidence that the planning of a grasping movement can modulate early visual cortex and provides a unifying framework for understanding the dual role played by the IPS in motor planning and attentional orienting.