Visual cues that are effective for contextual saccade adaptation

Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence

In the realm of robotics and automation, robot teleoperation, which facilitates human-machine interaction in distant or hazardous settings, has surged in significance. A persistent issue in this domain is the delays between command issuance and action execution, causing negative repercussions on operator situational awareness, performance, and cognitive load. These delays, particularly in long-distance operations, are difficult to mitigate even with the most advanced computing advancements. Current solutions mainly revolve around machine-based adjustments to combat these delays. However, a notable lacuna remains in harnessing human perceptions for an enhanced subjective teleoperation experience. This paper introduces a novel approach of sensory manipulation for induced human adaptation in delayed teleoperation. Drawing from motor learning and rehabilitation principles, it is posited that strategic sensory manipulation, via altered sensory stimuli, can mitigate the subjective feeling of these delays. The focus is not on introducing new skills or adapting to novel conditions; rather, it leverages prior motor coordination experience in the context of delays. The objective is to reduce the need for extensive training or sophisticated automation designs. A human-centered experiment involving 41 participants was conducted to examine the effects of modified haptic cues in teleoperations with delays. These cues were generated from high-fidelity physics engines using parameters from robot-end sensors or physics engine simulations. The results underscored several benefits, notably the considerable reduction in task time and enhanced user perceptions about visual delays. Real-time haptic feedback, or the anchoring method, emerged as a significant contributor to these benefits, showcasing reduced cognitive load, bolstered self-confidence, and minimized frustration. Beyond the prevalent methods of automation design and training, this research underscores induced human adaptation as a pivotal avenue in robot teleoperation. It seeks to enhance teleoperation efficacy through rapid human adaptation, offering insights beyond just optimizing robotic systems for delay compensations.

A High-Fidelity Agar-Based Phantom for Ultrasonography-Guided Brain Biopsy Simulation: A Novel Training Prototype with Visual Feedback

OBJECTIVE

A novel agar-based phantom was developed and assessed for ultrasonography (USG)-guided brain biopsy training. The phantom provides visual cues combined with sonologic cues, allowing multimodal training. Impact of multimodal training is evaluated through pretraining and posttraining trials.

METHODS

Twenty-five participants were divided based on experience with USG-based procedures into familiar (≥3 procedures performed in the past) (n = 14) and unfamiliar (<3 procedures performed) (n = 11). Agar phantoms with an opaque top and transparent middle layer were constructed in transparent glass bowls, each having 12 embedded targets. Participants underwent 2 supervised trials of USG-guided biopsy with aluminum foil covering the glass bowls, eliminating visual cues. Between 2 trials, participants underwent unsupervised self-training on a phantom without foil cover, providing visual cues. Performance was measured through insonation efficiency (EfI), biopsy efficiency (EfB), efficiency score (Ef), error score (Er), and performance score (PS). Scores were compared between and within the 2 groups before and after training. Impact of the self-training session on subjective comfort levels with the procedure was assessed through feedback forms.

RESULTS

Familiars had better pretraining EfB, Ef, Er, and PS (P < 0.001) compared with unfamiliars. After training, both performed similarly on all metrics. After training, familiars improved only in EfI (P = 0.001), with the unfamiliars showing significance in all metrics except EfI.

CONCLUSIONS

Simulation and phantom-based models can never supplant training through supervised skill application in vivo but our model supplements training by enabling technical skill acquisition, especially for beginners in USG-guided brain biopsy.

Contextual saccade adaptation induced by sequential saccades

Saccade adaptation is the gradual adjustment of saccade end point to maintain spatial accuracy. Contextual adaptation refers to a situation in which the adaptation-related change in saccade end point is contingent on the behavioral context in which the saccade is made. For example, in some situations, the same saccade to the same retinotopic location can be simultaneously adapted in opposite directions depending on the context in which it is made. Saccade adaptation has traditionally been studied in isolated movements, but in everyday life, saccades are often planned and executed in sequences. The oculomotor system may therefore have adaptive mechanisms specific to sequential saccades. Here, in five experiments, we investigated contextual saccade adaptation in sequences of saccades. In the first experiment, we demonstrate that saccades to a given retinotopic location can be simultaneously adapted in opposite directions depending on whether they occur in isolation or in a sequence. In the other experiments, we measured the extent to which properties of the previous and following saccades in a sequence can induce contextual saccade adaptation. Overall, we find that the existence, direction, and amplitude of previous and subsequent saccades, as well as the order of the current saccade within a movement sequence, can all induce contextual adaptation. These novel findings demonstrate the surprising flexibility of the system in maintaining end point accuracy, and support the idea that saccades made in a movement sequence are planned concurrently rather than independently.NEW & NOTEWORTHY This study reveals a new type of contextual saccade adaptation: sequential saccades are able to induce contextual saccade adaptation when direction, amplitude, or the existence of preceding and following saccades are used as contexts. These novel findings are also consistent with the idea that saccades made in a sequence are planned concurrently rather than independently.

A generative learning model for saccade adaptation

Plasticity in the oculomotor system ensures that saccadic eye movements reliably meet their visual goals-to bring regions of interest into foveal, high-acuity vision. Here, we present a comprehensive description of sensorimotor learning in saccades. We induced continuous adaptation of saccade amplitudes using a double-step paradigm, in which participants saccade to a peripheral target stimulus, which then undergoes a surreptitious, intra-saccadic shift (ISS) as the eyes are in flight. In our experiments, the ISS followed a systematic variation, increasing or decreasing from one saccade to the next as a sinusoidal function of the trial number. Over a large range of frequencies, we confirm that adaptation gain shows (1) a periodic response, reflecting the frequency of the ISS with a delay of a number of trials, and (2) a simultaneous drift towards lower saccade gains. We then show that state-space-based linear time-invariant systems (LTIS) represent suitable generative models for this evolution of saccade gain over time. This state-equation algorithm computes the prediction of an internal (or hidden state-) variable by learning from recent feedback errors, and it can be compared to experimentally observed adaptation gain. The algorithm also includes a forgetting rate that quantifies per-trial leaks in the adaptation gain, as well as a systematic, non-error-based bias. Finally, we study how the parameters of the generative models depend on features of the ISS. Driven by a sinusoidal disturbance, the state-equation admits an exact analytical solution that expresses the parameters of the phenomenological description as functions of those of the generative model. Together with statistical model selection criteria, we use these correspondences to characterize and refine the structure of compatible state-equation models. We discuss the relation of these findings to established results and suggest that they may guide further design of experimental research across domains of sensorimotor adaptation.

Discriminative control of saccade latencies

Recent studies have demonstrated that saccadic reaction times (SRTs) are influenced by the temporal regularities of dynamic environments (Vullings & Madelain, 2018). Here, we ask whether discriminative control (i.e., the possibility to use external stimuli signaling the future state of the environment) of latencies in a search task might be established using reinforcement contingencies. Eight participants made saccades within 80-750 ms toward a target displayed among distractors. We constructed two latency classes, "short" and "long," using the first and last quartiles of the individual baseline distributions. We then used a latency-contingent display paradigm in which finding the visual target among other items was made contingent upon specific SRTs. For a first group, the postsaccadic target was displayed only following short latencies with leftward saccades, and following long latencies with rightward saccades. The opposite was true for a second group. When short- and long-latency saccades were reinforced (i.e., the target was displayed) depending on the saccade direction, median latencies differed by 74 ms on average (all outside the 98% null hypothesis confidence intervals). Posttraining, in the absence of reinforcement, we still observed strong differences in latency distributions, averaging 64 ms for leftward versus rightward saccades. Our results demonstrate the discriminative control of SRTs, further supporting the effects of reinforcement learning for saccade. This study reveals that saccade triggering is finely controlled by learned temporal and spatial properties of the environment using predictive mechanisms.

Inter-individual variability and consistency of saccade adaptation in oblique saccades: Amplitude increase and decrease in the horizontal or vertical saccade component

Despite changes in the physical structures controlling the eyes, saccades, the rapid eye movements used to explore the visual environment, remain accurate throughout the lifetime. The process underlying this sensorimotor adaptation is studied using a double step paradigm: an intra-saccadic target displacement introduces a systematic position error which triggers changes in saccadic amplitude or direction across trials. Numerous researches on this saccade adaptation have been conducted, but the level of inter-individual variability and consistency in saccade gain change and how it relates to increase- or decrease-amplitude paradigms is not fully described. We conducted experiments in four groups of 25 participants with 800 trials per participant, including 200 baseline trials and 200 recovery trials. We used four distinct double-step paradigms that differed by the intra-saccadic target-step leading to either a horizontal (Backward or Forward) or vertical (Upward or Downward) gain modulation. Across experiments 95% of the participants exhibited adaptation, revealing the consistency of this phenomenon. We observed strong inter-individual differences, both in the extent and rate of adaptation, which were not correlated with the individual baseline saccades characteristics. As previously reported, the rates of adaptation were higher for gain decrease than for gain increase experiments but the final extent of adaptation were similar. Our results also support the view that adaptation of oblique saccades occurs where the saccade command is represented as a vector. Finally, at the individual level, we did not observe systematic changes in the saccade metrics in relation to adaptation.

The Effects of Sensory Manipulations on Motor Behavior: From Basic Science to Clinical Rehabilitation

Modifying sensory aspects of the learning environment can influence motor behavior. Although the effects of sensory manipulations on motor behavior have been widely studied, there still remains a great deal of variability across the field in terms of how sensory information has been manipulated or applied. Here, the authors briefly review and integrate the literature from each sensory modality to gain a better understanding of how sensory manipulations can best be used to enhance motor behavior. Then, they discuss 2 emerging themes from this literature that are important for translating sensory manipulation research into effective interventions. Finally, the authors provide future research directions that may lead to enhanced efficacy of sensory manipulations for motor learning and rehabilitation.

Error compensation in random vector double step saccades with and without global adaptation

In saccade sequences without visual feedback endpoint errors pose a problem for subsequent saccades. Accurate error compensation has previously been demonstrated in double step saccades (DSS) and is thought to rely on a copy of the saccade motor vector. However, these studies typically use fixed target vectors on each trial, calling into question the generalizability of the findings due to the high stimulus predictability. We present a random walk DSS paradigm (random target vector amplitudes and directions) to provide a more complete, realistic and generalizable description of error compensation in saccade sequences. We regressed the vector between the endpoint of the second saccade and the endpoint of a hypothetical second saccade that does not take first saccade error into account on the ideal compensation vector. This provides a direct and complete estimation of error compensation in DSS. We observed error compensation with varying stimulus displays that was comparable to previous findings. We also employed this paradigm to extend experiments that showed accurate compensation for systematic undershoots after specific-vector saccade adaptation. Utilizing the random walk paradigm for saccade adaptation by Rolfs et al. (2010) together with our random walk DSS paradigm we now also demonstrate transfer of adaptation from reactive to memory guided saccades for global saccade adaptation. We developed a new, generalizable DSS paradigm with unpredictable stimuli and successfully employed it to verify, replicate and extend previous findings, demonstrating that endpoint errors are compensated for saccades in all directions and variable amplitudes.

Saccadic adaptation to a systematically varying disturbance

Saccadic adaptation maintains the correct mapping between eye movements and their targets, yet the dynamics of saccadic gain changes in the presence of systematically varying disturbances has not been extensively studied. Here we assessed changes in the gain of saccade amplitudes induced by continuous and periodic postsaccadic visual feedback. Observers made saccades following a sequence of target steps either along the horizontal meridian (Two-way adaptation) or with unconstrained saccade directions (Global adaptation). An intrasaccadic step-following a sinusoidal variation as a function of the trial number (with 3 different frequencies tested in separate blocks)-consistently displaced the target along its vector. The oculomotor system responded to the resulting feedback error by modifying saccade amplitudes in a periodic fashion with similar frequency of variation but lagging the disturbance by a few tens of trials. This periodic response was superimposed on a drift toward stronger hypometria with similar asymptotes and decay rates across stimulus conditions. The magnitude of the periodic response decreased with increasing frequency and was smaller and more delayed for Global than Two-way adaptation. These results suggest that-in addition to the well-characterized return-to-baseline response observed in protocols using constant visual feedback-the oculomotor system attempts to minimize the feedback error by integrating its variation across trials. This process resembles a convolution with an internal response function, whose structure would be determined by coefficients of the learning model. Our protocol reveals this fast learning process in single short experimental sessions, qualifying it for the study of sensorimotor learning in health and disease.

Saccadic Adaptation in 10-41 Month-Old Children

When saccade amplitude becomes systematically inaccurate, adaptation mechanisms gradually decrease or increase it until accurate saccade targeting is recovered. Adaptive shortening and adaptive lengthening of saccade amplitude rely on separate mechanisms in adults. When these adaptation mechanisms emerge during development is poorly known except that adaptive shortening processes are functional in children above 8 years of age. Yet, saccades in infants are consistently inaccurate (hypometric) as if adaptation mechanisms were not fully functional in early childhood. Here, we tested reactive saccade adaptation in 10–41 month-old children compared to a group of 20–30 year-old adults. A visual target representing a cartoon character appeared at successive and unpredictable locations 10° apart on a computer screen. During the eye movement toward the target, it systematically stepped in the direction opposite to the saccade to induce an adaptive shortening of saccade amplitude (Experiment 1). In Experiment 2, the target stepped in the same direction as the ongoing saccade to induce an adaptive lengthening of saccade amplitude. In both backward and forward adaptation experiments, saccade adaptation was compared to a control condition where there was no intrasaccadic target step. Analysis of baseline performance revealed both longer saccade reaction times and hypometric saccades in children compared to adults. In both experiments, children on average showed gradual changes in saccade amplitude consistent with the systematic intrasaccadic target steps. Moreover, the amount of amplitude change was similar between children and adults for both backward and forward adaptation. Finally, adaptation abilities in our child group were not related to age. Overall the results suggest that the neural mechanisms underlying reactive saccade adaptation are in place early during development.

Context cue-dependent saccadic adaptation in rhesus macaques cannot be elicited using color

When the head does not move, rapid movements of the eyes called saccades are used to redirect the line of sight. Saccades are defined by a series of metrical and kinematic (evolution of a movement as a function of time) relationships. For example, the amplitude of a saccade made from one visual target to another is roughly 90% of the distance between the initial fixation point (T0) and the peripheral target (T1). However, this stereotypical relationship between saccade amplitude and initial retinal error (T1-T0) may be altered, either increased or decreased, by surreptitiously displacing a visual target during an ongoing saccade. This form of motor learning (called saccadic adaptation) has been described in both humans and monkeys. Recent experiments in humans and monkeys have suggested that internal (proprioceptive) and external (target shape, color, and/or motion) cues may be used to produce context-dependent adaptation. We tested the hypothesis that an external contextual cue (target color) could be used to evoke differential gain (actual saccade/initial retinal error) states in rhesus monkeys. We did not observe differential gain states correlated with target color regardless of whether targets were displaced along the same vector as the primary saccade or perpendicular to it. Furthermore, this observation held true regardless of whether adaptation trials using various colors and intrasaccade target displacements were randomly intermixed or presented in short or long blocks of trials. These results are consistent with hypotheses that state that color cannot be used as a contextual cue and are interpreted in light of previous studies of saccadic adaptation in both humans and monkeys.