A brain machine interface framework for exploring proactive control of smart environments

Brain machine interfaces (BMIs) can substantially improve the quality of life of elderly or disabled people. However, performing complex action sequences with a BMI system is onerous because it requires issuing commands sequentially. Fundamentally different from this, we have designed a BMI system that reads out mental planning activity and issues commands in a proactive manner. To demonstrate this, we recorded brain activity from freely-moving monkeys performing an instructed task and decoded it with an energy-efficient, small and mobile field-programmable gate array hardware decoder triggering real-time action execution on smart devices. Core of this is an adaptive decoding algorithm that can compensate for the day-by-day neuronal signal fluctuations with minimal re-calibration effort. We show that open-loop planning-ahead control is possible using signals from primary and pre-motor areas leading to significant time-gain in the execution of action sequences. This novel approach provides, thus, a stepping stone towards improved and more humane control of different smart environments with mobile brain machine interfaces.

Position- and scale-invariant object-centered spatial localization in monkey frontoparietal cortex dynamically adapts to cognitive demand

Egocentric encoding is a well-known property of brain areas along the dorsal pathway. Different to previous experiments, which typically only demanded egocentric spatial processing during movement preparation, we designed a task where two male rhesus monkeys memorized an on-the-object target position and then planned a reach to this position after the object re-occurred at variable location with potentially different size. We found allocentric (in addition to egocentric) encoding in the dorsal stream reach planning areas, parietal reach region and dorsal premotor cortex, which is invariant with respect to the position, and, remarkably, also the size of the object. The dynamic adjustment from predominantly allocentric encoding during visual memory to predominantly egocentric during reach planning in the same brain areas and often the same neurons, suggests that the prevailing frame of reference is less a question of brain area or processing stream, but more of the cognitive demands.

Deciding while acting - Mid-movement decisions are more strongly affected by action probability than reward amount

When deciding while acting, such as sequentially selecting targets during naturalistic foraging, movement trajectories reveal the dynamics of the unfolding decision process. Ongoing and planned actions may impact decisions in these situations in addition to expected reward outcomes. Here, we test how strongly humans weigh and how fast they integrate individual constituents of expected value, namely the prior probability of an action (PROB) and the prior expected reward amount associated with an action (AMNT), when deciding based on the combination of both together during an ongoing movement. Unlike other decision-making studies, we focus on PROB and AMNT priors, and not final evidence, in that correct actions were either instructed or could be chosen freely. This means, there was no decision-making under risk. We show that both priors gradually influence movement trajectories already before mid-movement instructions of the correct target and bias free-choice behavior. These effects were consistently stronger for PROB compared to AMNT priors. Participants biased their movements towards a high-PROB target, committed to it faster when instructed or freely chosen, and chose it more frequently even when it was associated with a lower AMNT prior than the alternative option. Despite these differences in effect magnitude, the time course of both priors' effect on movement direction was highly similar. We conclude that prior action probability, and hence the associated possibility to plan actions accordingly, has higher behavioral relevance than prior action value for decisions that are expressed by adjusting already ongoing movements.Significance StatementNatural behavior, like foraging or hunting prey, requires animals and humans to select their next action during ongoing movements, thereby updating movements as the decision process unfolds. Here, we study the magnitude and time course with which prior action probability and prior expectancy of reward amount influence the selection between two competing movements in humans. By simultaneously but independently manipulating both priors in individual decisions, and by avoiding confounds of reward probability, we show that both priors affect the decision process with different magnitude yet comparable time courses. Our results emphasize the prioritized relevance of action probabilities over action values on mid-movement decisions.

Human and macaque pairs employ different coordination strategies in a transparent decision game

Many real-world decisions in social contexts are made while observing a partner's actions. To study dynamic interactions during such decisions, we developed a setup where two agents seated face-to-face to engage in game-theoretical tasks on a shared transparent touchscreen display ('transparent games'). We compared human and macaque pairs in a transparent version of the coordination game 'Bach-or-Stravinsky', which entails a conflict about which of two individually-preferred opposing options to choose to achieve coordination. Most human pairs developed coordinated behavior and adopted dynamic turn-taking to equalize the payoffs. All macaque pairs converged on simpler, static coordination. Remarkably, two animals learned to coordinate dynamically after training with a human confederate. This pair selected the faster agent's preferred option, exhibiting turn-taking behavior that was captured by modeling the visibility of the partner's action before one's own movement. Such competitive turn-taking was unlike the prosocial turn-taking in humans, who equally often initiated switches to and from their preferred option. Thus, the dynamic coordination is not restricted to humans but can occur on the background of different social attitudes and cognitive capacities in rhesus monkeys. Overall, our results illustrate how action visibility promotes the emergence and maintenance of coordination when agents can observe and time their mutual actions.

Turning decisions into actions

Are selection and control of action serial processes of separate neural modules? A new study in PLOS Biology argues against this and in favor of an integrated process distributed across multiple brain regions, each contributing in a distinct way.

The future of General Movement Assessment: The role of computer vision and machine learning - A scoping review

BACKGROUND

The clinical and scientific value of Prechtl general movement assessment (GMA) has been increasingly recognised, which has extended beyond the detection of cerebral palsy throughout the years. With advancing computer science, a surging interest in developing automated GMA emerges.

AIMS

In this scoping review, we focused on video-based approaches, since it remains authentic to the non-intrusive principle of the classic GMA. Specifically, we aimed to provide an overview of recent video-based approaches targeting GMs; identify their techniques for movement detection and classification; examine if the technological solutions conform to the fundamental concepts of GMA; and discuss the challenges of developing automated GMA.

METHODS AND PROCEDURES

We performed a systematic search for computer vision-based studies on GMs.

OUTCOMES AND RESULTS

We identified 40 peer-reviewed articles, most (n = 30) were published between 2017 and 2020. A wide variety of sensing, tracking, detection, and classification tools for computer vision-based GMA were found. Only a small portion of these studies applied deep learning approaches. A comprehensive comparison between data acquisition and sensing setups across the reviewed studies, highlighting limitations and advantages of each modality in performing automated GMA is provided.

CONCLUSIONS AND IMPLICATIONS

A "method-of-choice" for automated GMA does not exist. Besides creating large datasets, understanding the fundamental concepts and prerequisites of GMA is necessary for developing automated solutions. Future research shall look beyond the narrow field of detecting cerebral palsy and open up to the full potential of applying GMA to enable an even broader application.

An Open Resource for Non-human Primate Optogenetics

Optogenetics has revolutionized neuroscience in small laboratory animals, but its effect on animal models more closely related to humans, such as non-human primates (NHPs), has been mixed. To make evidence-based decisions in primate optogenetics, the scientific community would benefit from a centralized database listing all attempts, successful and unsuccessful, of using optogenetics in the primate brain. We contacted members of the community to ask for their contributions to an open science initiative. As of this writing, 45 laboratories around the world contributed more than 1,000 injection experiments, including precise details regarding their methods and outcomes. Of those entries, more than half had not been published. The resource is free for everyone to consult and contribute to on the Open Science Framework website. Here we review some of the insights from this initial release of the database and discuss methodological considerations to improve the success of optogenetic experiments in NHPs.

Wireless recording from unrestrained monkeys reveals motor goal encoding beyond immediate reach in frontoparietal cortex

System neuroscience of motor cognition regarding the space beyond immediate reach mandates free, yet experimentally controlled movements. We present an experimental environment (Reach Cage) and a versatile visuo-haptic interaction system (MaCaQuE) for investigating goal-directed whole-body movements of unrestrained monkeys. Two rhesus monkeys conducted instructed walk-and-reach movements towards targets flexibly positioned in the cage. We tracked 3D multi-joint arm and head movements using markerless motion capture. Movements show small trial-to-trial variability despite being unrestrained. We wirelessly recorded 192 broad-band neural signals from three cortical sensorimotor areas simultaneously. Single unit activity is selective for different reach and walk-and-reach movements. Walk-and-reach targets could be decoded from premotor and parietal but not motor cortical activity during movement planning. The Reach Cage allows systems-level sensorimotor neuroscience studies with full-body movements in a configurable 3D spatial setting with unrestrained monkeys. We conclude that the primate frontoparietal network encodes reach goals beyond immediate reach during movement planning.

Experimental Testing of Bionic Peripheral Nerve and Muscle Interfaces: Animal Model Considerations

Introduction: Man-machine interfacing remains the main challenge for accurate and reliable control of bionic prostheses. Implantable electrodes in nerves and muscles may overcome some of the limitations by significantly increasing the interface's reliability and bandwidth. Before human application, experimental preclinical testing is essential to assess chronic in-vivo biocompatibility and functionality. Here, we analyze available animal models, their costs and ethical challenges in special regards to simulating a potentially life-long application in a short period of time and in non-biped animals.

Methods: We performed a literature analysis following the PRISMA guidelines including all animal models used to record neural or muscular activity via implantable electrodes, evaluating animal models, group size, duration, origin of publication as well as type of interface. Furthermore, behavioral, ethical, and economic considerations of these models were analyzed. Additionally, we discuss experience and surgical approaches with rat, sheep, and primate models and an approach for international standardized testing.

Results: Overall, 343 studies matched the search terms, dominantly originating from the US (55%) and Europe (34%), using mainly small animal models (rat: 40%). Electrode placement was dominantly neural (77%) compared to muscular (23%). Large animal models had a mean duration of 135 ± 87.2 days, with a mean of 5.3 ± 3.4 animals per trial. Small animal models had a mean duration of 85 ± 11.2 days, with a mean of 12.4 ± 1.7 animals.

Discussion: Only 37% animal models were by definition chronic tests (>3 months) and thus potentially provide information on long-term performance. Costs for large animals were up to 45 times higher than small animals. However, costs are relatively small compared to complication costs in human long-term applications. Overall, we believe a combination of small animals for preliminary primary electrode testing and large animals to investigate long-term biocompatibility, impedance, and tissue regeneration parameters provides sufficient data to ensure long-term human applications.

Emergence and suppression of cooperation by action visibility in transparent games

Real-world agents, humans as well as animals, observe each other during interactions and choose their own actions taking the partners' ongoing behaviour into account. Yet, classical game theory assumes that players act either strictly sequentially or strictly simultaneously without knowing each other's current choices. To account for action visibility and provide a more realistic model of interactions under time constraints, we introduce a new game-theoretic setting called transparent games, where each player has a certain probability of observing the partner's choice before deciding on its own action. By means of evolutionary simulations, we demonstrate that even a small probability of seeing the partner's choice before one's own decision substantially changes the evolutionary successful strategies. Action visibility enhances cooperation in an iterated coordination game, but reduces cooperation in a more competitive iterated Prisoner's Dilemma. In both games, "Win-stay, lose-shift" and "Tit-for-tat" strategies are predominant for moderate transparency, while a "Leader-Follower" strategy emerges for high transparency. Our results have implications for studies of human and animal social behaviour, especially for the analysis of dyadic and group interactions.

Complementary encoding of priors in monkey frontoparietal network supports a dual process of decision-making

Prior expectations of movement instructions can promote preliminary action planning and influence choices. We investigated how action priors affect action-goal encoding in premotor and parietal cortices and if they bias subsequent free choice. Monkeys planned reaches according to visual cues that indicated relative probabilities of two possible goals. On instructed trials, the reach goal was determined by a secondary cue respecting these probabilities. On rarely interspersed free-choice trials without instruction, both goals offered equal reward. Action priors induced graded free-choice biases and graded frontoparietal motor-goal activity, complementarily in two subclasses of neurons. Down-regulating neurons co-encoded both possible goals and decreased opposite-to-preferred responses with decreasing prior, possibly supporting a process of choice by elimination. Up-regulating neurons showed increased preferred-direction responses with increasing prior, likely supporting a process of computing net likelihood. Action-selection signals emerged earliest in down-regulating neurons of premotor cortex, arguing for an initiation of selection in the frontal lobe.

Directed Interaction Between Monkey Premotor and Posterior Parietal Cortex During Motor-Goal Retrieval from Working Memory

Goal-directed behavior requires cognitive control of action, putatively by means of frontal-lobe impact on posterior brain areas. We investigated frontoparietal directed interaction (DI) in monkeys during memory-guided rule-based reaches, to test if DI supports motor-goal selection or working memory (WM) processes. We computed DI between the parietal reach region (PRR) and dorsal premotor cortex (PMd) with a Granger-causality measure of intracortical local field potentials (LFP). LFP mostly in the beta (12–32 Hz) and low-frequency (f≤10Hz) ranges contributed to DI. During movement withholding, beta-band activity in PRR had a Granger-causal effect on PMd independent of WM content. Complementary, low-frequency PMd activity had a transient Granger-causing effect on PRR specifically during WM retrieval of spatial motor goals, while no DI was associated with preliminary motor-goal selection. Our results support the idea that premotor and posterior parietal cortices interact functionally to achieve cognitive control during goal-directed behavior, in particular, that frontal-to-parietal interaction occurs during retrieval of motor-goal information from spatial WM.

Comparing Open-Source Toolboxes for Processing and Analysis of Spike and Local Field Potentials Data

Analysis of spike and local field potential (LFP) data is an essential part of neuroscientific research. Today there exist many open-source toolboxes for spike and LFP data analysis implementing various functionality. Here we aim to provide a practical guidance for neuroscientists in the choice of an open-source toolbox best satisfying their needs. We overview major open-source toolboxes for spike and LFP data analysis as well as toolboxes with tools for connectivity analysis, dimensionality reduction and generalized linear modeling. We focus on comparing toolboxes functionality, statistical and visualization tools, documentation and support quality. To give a better insight, we compare and illustrate functionality of the toolboxes on open-access dataset or simulated data and make corresponding MATLAB scripts publicly available.

Peri-hand space expands beyond reach in the context of walk-and-reach movements

The brain incorporates sensory information across modalities to be able to interact with our environment. The peripersonal space (PPS), defined by a high level of crossmodal interaction, is centered on the relevant body part, e.g. the hand, but can spatially expand to encompass tools or reach targets during goal-directed behavior. Previous studies considered expansion of the PPS towards goals within immediate or tool-mediated reach, but not the translocation of the body as during walking. Here, we used the crossmodal congruency effect (CCE) to quantify the extension of the PPS and test if PPS can also expand further to include far located walk-and-reach targets accessible only by translocation of the body. We tested for orientation specificity of the hand-centered reference frame, asking if the CCE inverts with inversion of the hand orientation during reach. We show a high CCE with onset of the movement not only towards reach targets but also walk-and-reach targets. When participants must change hand orientation, the CCE decreases, if not vanishes, and does not simply invert. We conclude that the PPS can expand to the action space beyond immediate or tool-mediated reaching distance but is not purely hand-centered with respect to orientation.

Correction: Using imaging photoplethysmography for heart rate estimation in non-human primates

[This corrects the article DOI: 10.1371/journal.pone.0202581.].

Using imaging photoplethysmography for heart rate estimation in non-human primates

For humans and for non-human primates heart rate is a reliable indicator of an individual's current physiological state, with applications ranging from health checks to experimental studies of cognitive and emotional state. In humans, changes in the optical properties of the skin tissue correlated with cardiac cycles (imaging photoplethysmogram, iPPG) allow non-contact estimation of heart rate by its proxy, pulse rate. Yet, there is no established simple and non-invasive technique for pulse rate measurements in awake and behaving animals. Using iPPG, we here demonstrate that pulse rate in rhesus monkeys can be accurately estimated from facial videos. We computed iPPGs from eight color facial videos of four awake head-stabilized rhesus monkeys. Pulse rate estimated from iPPGs was in good agreement with reference data from a contact pulse-oximeter: the error of pulse rate estimation was below 5% of the individual average pulse rate in 83% of the epochs; the error was below 10% for 98% of the epochs. We conclude that iPPG allows non-invasive and non-contact estimation of pulse rate in non-human primates, which is useful for physiological studies and can be used toward welfare-assessment of non-human primates in research.

Implantable computer-controlled adaptive multielectrode positioning system

Acute neuronal recordings performed with metal microelectrodes in nonhuman primates allow investigating the neural substrate of complex cognitive behaviors. Yet the daily reinsertion and positioning of the electrodes prevents recording from many neurons simultaneously, limiting the suitability of these types of recordings for brain-computer interface applications or for large-scale population statistical methods on a trial-by-trial basis. In contrast, chronically implanted multielectrode arrays offer the opportunity to record from many neurons simultaneously, but immovable electrodes prevent optimization of the signal during and after implantation and cause the tissue response to progressively impair the transduced signal quality, thereby limiting the number of different neurons that can be recorded over the lifetime of the implant. Semichronically implanted matrices of electrodes, instead, allow individually movable electrodes in depth and achieve higher channel count compared with acute methods, hence partially overcoming these limitations. Existing semichronic systems with higher channel count lack computerized control of electrode movements, leading to limited user-friendliness and uncertainty in depth positioning. Here we demonstrate a chronically implantable adaptive multielectrode positioning system with detachable drive for computerized depth adjustment of individual electrodes over several millimeters. This semichronic 16-channel system is designed to optimize the simultaneous yield of units in an extended period following implantation since the electrodes can be independently depth adjusted with minimal effort and their signal quality continuously assessed. Importantly, the electrode array is designed to remain within a chronic recording chamber for a prolonged time or can be used for acute recordings with high signal-to-noise ratio in the cerebral cortex of nonhuman primates.

NEW & NOTEWORTHY We present a 16-channel motorized, semichronic multielectrode array with individually depth-adjustable electrodes to record in the cerebral cortex of nonhuman primates. Compared with fixed-geometry arrays, this system allows repeated reestablishing of single neuron isolation. Compared with manually adjustable arrays it benefits from computer-controlled positioning. Compared with motorized semichronic systems it allows higher channel counts due to a robotic single actuator approach. Overall the system is designed to optimize the simultaneous yield of units over the course of implantation.

Standardized automated training of rhesus monkeys for neuroscience research in their housing environment

Teaching nonhuman primates the complex cognitive behavioral tasks that are central to cognitive neuroscience research is an essential and challenging endeavor. It is crucial for the scientific success that the animals learn to interpret the often complex task rules and reliably and enduringly act accordingly. To achieve consistent behavior and comparable learning histories across animals, it is desirable to standardize training protocols. Automatizing the training can significantly reduce the time invested by the person training the animal. In addition, self-paced training schedules with individualized learning speeds based on automatic updating of task conditions could enhance the animals’ motivation and welfare. We developed a training paradigm for across-task unsupervised training (AUT) of successively more complex cognitive tasks to be administered through a stand-alone housing-based system optimized for rhesus monkeys in neuroscience research settings (Calapai A, Berger M, Niessing M, Heisig K, Brockhausen R, Treue S, Gail A. Behav Res Methods 5: 1–11, 2016). The AUT revealed interindividual differences in long-term learning progress between animals, helping to characterize learning personalities, and commonalities, helping to identify easier and more difficult learning steps in the training protocol. Our results demonstrate that 1) rhesus monkeys stay engaged with the AUT over months despite access to water and food outside the experimental sessions but with lower numbers of interaction compared with conventional fluid-controlled training; 2) with unsupervised training across sessions and task levels, rhesus monkeys can learn tasks of sufficient complexity for state-of-the-art cognitive neuroscience in their housing environment; and 3) AUT learning progress is primarily determined by the number of interactions with the system rather than the mere exposure time.

NEW & NOTEWORTHY We demonstrate that highly structured training of behavioral tasks, as used in neuroscience research, can be achieved in an unsupervised fashion over many sessions and task difficulties in a monkey housing environment. Employing a predefined training strategy allows for an observer-independent comparison of learning between animals and of training approaches. We believe that self-paced standardized training can be utilized for pretraining and animal selection and can contribute to animal welfare in a neuroscience research environment.

Assessment of stress responses in rhesus macaques (Macaca mulatta) to daily routine procedures in system neuroscience based on salivary cortisol concentrations

Non-human primates participating in neurophysiological research are exposed to potentially stressful experimental procedures, such as dietary control protocols, surgical implants and their maintenance, or social separation during training and experimental session. Here, we investigated the effect of controlled access to fluid, surgical implants, implant-related cleaning of skin margins, and behavioral training sessions on salivary cortisol levels of adult male rhesus macaques participating in neurophysiological research. The animals were trained to chew flavored cotton swabs to non-invasively collect saliva samples. Our data show no differences in cortisol levels between animals with and without implants, but both, controlled access to fluid and cleaning of implants individually increased salivary cortisol concentrations, while both together did not further increase the concentration. Specifically, before cleaning, individuals with controlled access to fluid had 55% higher cortisol concentrations than individuals with free access to fluid. Under free access to fluid, cortisol concentrations were 27% higher after cleaning while no effect of cleaning was found for individuals under controlled fluid access. Training sessions under controlled access to fluid also did not affect salivary cortisol concentrations. The observed changes in cortisol concentrations represent mild stress responses, as they are only a fraction of the range of the regular circadian changes in cortisol levels in rhesus monkeys. They also indicate that combinations of procedures do not necessarily lead to cumulative stress responses. Our results indicate that salivary cortisol levels of rhesus monkeys respond to neurophysiological experimental procedures and, hence, may be used to assess further refinements of such experimental methods.

What makes a reach movement effortful? Physical effort discounting supports common minimization principles in decision making and motor control

When deciding between alternative options, a rational agent chooses on the basis of the desirability of each outcome, including associated costs. As different options typically result in different actions, the effort associated with each action is an essential cost parameter. How do humans discount physical effort when deciding between movements? We used an action-selection task to characterize how subjective effort depends on the parameters of arm transport movements and controlled for potential confounding factors such as delay discounting and performance. First, by repeatedly asking subjects to choose between 2 arm movements of different amplitudes or durations, performed against different levels of force, we identified parameter combinations that subjects experienced as identical in effort (isoeffort curves). Movements with a long duration were judged more effortful than short-duration movements against the same force, while movement amplitudes did not influence effort. Biomechanics of the movements also affected effort, as movements towards the body midline were preferred to movements away from it. Second, by introducing movement repetitions, we further determined that the cost function for choosing between effortful movements had a quadratic relationship with force, while choices were made on the basis of the logarithm of these costs. Our results show that effort-based action selection during reaching cannot easily be explained by metabolic costs. Instead, force-loaded reaches, a widely occurring natural behavior, imposed an effort cost for decision making similar to cost functions in motor control. Our results thereby support the idea that motor control and economic choice are governed by partly overlapping optimization principles.

Economic choice in humans and animals can be understood as a weighing of benefits (e.g., reward) against costs (e.g., effort, delay, risk), leading to a preference for the behavioral option with highest expected utility. The costs of the action associated with a choice can thereby affect its utility: for equivalent benefits, an action that requires less physical effort will be preferred to a more effortful one. Here, we characterized how human subjects assess physical effort when choosing between arm movements. We show that the effort cost of a movement increases with its duration and with the square of the force it is performed against but not with the distance covered. Therefore, the subjective cost that determines decisions does not reflect the objective energetic cost of the actions—i.e., the corresponding metabolic expenditure. Instead, the subjective cost has commonalities with the cost that our central nervous system is believed to minimize for controlling the motor execution of actions. Our findings thus argue in favor of action selection and action control sharing common underlying optimization principles.

I Plan Therefore I Choose: Free-Choice Bias Due to Prior Action-Probability but Not Action-Value

According to an emerging view, decision-making, and motor planning are tightly entangled at the level of neural processing. Choice is influenced not only by the values associated with different options, but also biased by other factors. Here we test the hypothesis that preliminary action planning can induce choice biases gradually and independently of objective value when planning overlaps with one of the potential action alternatives. Subjects performed center-out reaches obeying either a clockwise or counterclockwise cue-response rule in two tasks. In the probabilistic task, a pre-cue indicated the probability of each of the two potential rules to become valid. When the subsequent rule-cue unambiguously indicated which of the pre-cued rules was actually valid (instructed trials), subjects responded faster to rules pre-cued with higher probability. When subjects were allowed to choose freely between two equally rewarded rules (choice trials) they chose the originally more likely rule more often and faster, despite the lack of an objective advantage in selecting this target. In the amount task, the pre-cue indicated the amount of potential reward associated with each rule. Subjects responded faster to rules pre-cued with higher reward amount in instructed trials of the amount task, equivalent to the more likely rule in the probabilistic task. Yet, in contrast, subjects showed hardly any choice bias and no increase in response speed in favor of the original high-reward target in the choice trials of the amount task. We conclude that free-choice behavior is robustly biased when predictability encourages the planning of one of the potential responses, while prior reward expectations without action planning do not induce such strong bias. Our results provide behavioral evidence for distinct contributions of expected value and action planning in decision-making and a tight interdependence of motor planning and action selection, supporting the idea that the underlying neural mechanisms overlap.

Asymmetric generalization in adaptation to target displacement errors in humans and in a neural network model

Different error signals can induce sensorimotor adaptation during visually guided reaching, possibly evoking different neural adaptation mechanisms. Here we investigate reach adaptation induced by visual target errors without perturbing the actual or sensed hand position. We analyzed the spatial generalization of adaptation to target error to compare it with other known generalization patterns and simulated our results with a neural network model trained to minimize target error independent of prediction errors. Subjects reached to different peripheral visual targets and had to adapt to a sudden fixed-amplitude displacement ("jump") consistently occurring for only one of the reach targets. Subjects simultaneously had to perform contralateral unperturbed saccades, which rendered the reach target jump unnoticeable. As a result, subjects adapted by gradually decreasing reach errors and showed negative aftereffects for the perturbed reach target. Reach errors generalized to unperturbed targets according to a translational rather than rotational generalization pattern, but locally, not globally. More importantly, reach errors generalized asymmetrically with a skewed generalization function in the direction of the target jump. Our neural network model reproduced the skewed generalization after adaptation to target jump without having been explicitly trained to produce a specific generalization pattern. Our combined psychophysical and simulation results suggest that target jump adaptation in reaching can be explained by gradual updating of spatial motor goal representations in sensorimotor association networks, independent of learning induced by a prediction-error about the hand position. The simulations make testable predictions about the underlying changes in the tuning of sensorimotor neurons during target jump adaptation.

Planning Movements in Visual and Physical Space in Monkey Posterior Parietal Cortex

Neurons in the posterior parietal cortex respond selectively for spatial parameters of planned goal-directed movements. Yet, it is still unclear which aspects of the movement the neurons encode: the spatial parameters of the upcoming physical movement (physical goal), or the upcoming visual limb movement (visual goal). To test this, we recorded neuronal activity from the parietal reach region while monkeys planned reaches under either normal or prism-reversed viewing conditions. We found predominant encoding of physical goals while fewer neurons were selective for visual goals during planning. In contrast, local field potentials recorded in the same brain region exhibited predominant visual goal encoding, similar to previous imaging data from humans. The visual goal encoding in individual neurons was neither related to immediate visual input nor to visual memory, but to the future visual movement. Our finding suggests that action planning in parietal cortex is not exclusively a precursor of impending physical movements, as reflected by the predominant physical goal encoding, but also contains spatial kinematic parameters of upcoming visual movement, as reflected by co-existing visual goal encoding in neuronal spiking. The co-existence of visual and physical goals adds a complementary perspective to the current understanding of parietal spatial computations in primates.

Synchronization patterns suggest different functional organization in parietal reach region and dorsal premotor cortex

The parietal reach region (PRR) and dorsal premotor cortex (PMd) form part of the fronto-parietal reach network. While neural selectivity profiles of single-cell activity in these areas can be remarkably similar, other data suggest that both areas serve different computational functions in visually guided reaching. Here we test the hypothesis that different neural functional organizations characterized by different neural synchronization patterns might be underlying the putatively different functional roles. We use cross-correlation analysis on single-unit activity (SUA) and multiunit activity (MUA) to determine the prevalence of synchronized neural ensembles within each area. First, we reliably find synchronization in PRR but not in PMd. Second, we demonstrate that synchronization in PRR is present in different cognitive states, including "idle" states prior to task-relevant instructions and without neural tuning. Third, we show that local field potentials (LFPs) in PRR but not PMd are characterized by an increased power and spike field coherence in the beta frequency range (12-30 Hz), further indicating stronger synchrony in PRR compared with PMd. Finally, we show that neurons with similar tuning properties tend to be correlated in their random spike rate fluctuations in PRR but not in PMd. Our data support the idea that PRR and PMd, despite striking similarity in single-cell tuning properties, are characterized by unequal local functional organization, which likely reflects different network architectures to support different functional roles within the fronto-parietal reach network.

When adaptive control fails: Slow recovery of reduced rapid online control during reaching under reversed vision

Previous studies have shown that short-term exposure to mirror-reversed visual feedback suppresses rapid online control (ROC) of arm movements in response to a sudden target displacement. Here we tested if the reduced ROC under reversed vision can be observed for natural reaches without target perturbations, i.e. without corrective movements that are driven by visual input perturbation. Second, we ask if such ROC reduction generalizes to movement phases without visual feedback of the hand. Subjects were instructed to perform simple reach movements towards a stationary target position either under normal or physically reversed vision of the hand during the late movement phase. We quantified time-resolved ROC via a coefficient of determination of the reach trajectories over the full course of the movement. As for other measures in previous studies, we found that our perturbation-independent ROC was reduced within a few trials after exposure to reversed visual feedback. The reduced ROC was restricted to late movement phases, and was not observed in early movement phases. We further asked if subjects would be able to re-gain ROC with prolonged exposure to the reversed visual input. ROC gradually and incompletely increased over the course of 400 exposure trials, affecting both early and late movement phases. Our results show that under reversed vision ROC is reduced even for perturbation-independent natural reaches aiming at stationary targets.

Spatial task context makes short-latency reaches prone to induced Roelofs illusion

The perceptual localization of an object is often more prone to illusions than an immediate visuomotor action towards that object. The induced Roelofs effect (IRE) probes the illusory influence of task-irrelevant visual contextual stimuli on the processing of task-relevant visuospatial instructions during movement preparation. In the IRE, the position of a task-irrelevant visual object induces a shift in the localization of a visual target when subjects indicate the position of the target by verbal response, key-presses or delayed pointing to the target (“perception” tasks), but not when immediately pointing or reaching towards it without instructed delay (“action” tasks). This discrepancy was taken as evidence for the dual-visual-stream or perception-action hypothesis, but was later explained by a phasic distortion of the egocentric spatial reference frame which is centered on subjective straight-ahead (SSA) and used for reach planning. Both explanations critically depend on delayed movements to explain the IRE for action tasks. Here we ask: first, if the IRE can be observed for short-latency reaches; second, if the IRE in fact depends on a distorted egocentric frame of reference. Human subjects were tested in new versions of the IRE task in which the reach goal had to be localized with respect to another object, i.e., in an allocentric reference frame. First, we found an IRE even for immediate reaches in our allocentric task, but not for an otherwise similar egocentric control task. Second, the IRE depended on the position of the task-irrelevant frame relative to the reference object, not relative to SSA. We conclude that the IRE for reaching does not mandatorily depend on prolonged response delays, nor does it depend on motor planning in an egocentric reference frame. Instead, allocentric encoding of a movement goal is sufficient to make immediate reaches susceptible to IRE, underlining the context dependence of visuomotor illusions.

Choosing goals, not rules: deciding among rule-based action plans

In natural situations, movements are often directed toward locations different from that of the evoking sensory stimulus. Movement goals must then be inferred from the sensory cue based on rules. When there is uncertainty about the rule that applies for a given cue, planning a movement involves both choosing the relevant rule and computing the movement goal based on that rule. Under these conditions, it is not clear whether primates compute multiple movement goals based on all possible rules before choosing an action, or whether they first choose a rule and then only represent the movement goal associated with that rule. Supporting the former hypothesis, we show that neurons in the frontoparietal reach areas of monkeys simultaneously represent two different rule-based movement goals, which are biased by the monkeys' choice preferences. Apparently, primates choose between multiple behavioral options by weighing against each other the movement goals associated with each option.

What is 'anti' about anti-reaches? Reference frames selectively affect reaction times and endpoint variability

Reach movement planning involves the representation of spatial target information in different reference frames. Neurons at parietal and premotor stages of the cortical sensorimotor system represent target information in eye- or hand-centered reference frames, respectively. How the different neuronal representations affect behavioral parameters of motor planning and control, i.e. which stage of neural representation is relevant for which aspect of behavior, is not obvious from the physiology. Here, we test with a behavioral experiment if different kinematic movement parameters are affected to a different degree by either an eye- or hand-reference frame. We used a generalized anti-reach task to test the influence of stimulus-response compatibility (SRC) in eye- and hand-reference frames on reach reaction times, movement times, and endpoint variability. While in a standard anti-reach task, the SRC is identical in the eye- and hand-reference frames, we could separate SRC for the two reference frames. We found that reaction times were influenced by the SRC in eye- and hand-reference frame. In contrast, movement times were only influenced by the SRC in hand-reference frame, and endpoint variability was only influenced by the SRC in eye-reference frame. Since movement time and endpoint variability are the result of planning and control processes, while reaction times are consequences of only the planning process, we suggest that SRC effects on reaction times are highly suited to investigate reference frames of movement planning, and that eye- and hand-reference frames have distinct effects on different phases of motor action and different kinematic movement parameters.

Acquisition of myoelectric signals to control a hand prosthesis with implantable epimysial electrodes

The acquisition of myoelectric signals from the Musculus deltoideus of a rhesus monkey is described. Such signals are aimed to be used as control signal for an active myoelectric hand prosthesis. For recording, implantable flexible, polyimide-based multi-site microelectrodes were placed epimysially on the muscle. EMG signals were recorded during voluntary goal-directed movements of the arm, and analyzed with respect to signal amplitude and frequency.

Gain mechanisms for contextually guided visuomotor transformations

A prevailing question in sensorimotor research is the integration of sensory signals with abstract behavioral rules (contexts) and how this results in decisions about motor actions. We used neural network models to study how context-specific visuomotor remapping may depend on the functional connectivity among multiple layers. Networks were trained to perform different rotational visuomotor associations, depending on the stimulus color (a nonspatial context signal). In network I, the context signal was propagated forward through the network (bottom-up), whereas in network II, it was propagated backwards (top-down). During the presentation of the visual cue stimulus, both networks integrate the context with the sensory information via a mechanism similar to the classic gain field. The recurrence in the networks hidden layers allowed a simulation of the multimodal integration over time. Network I learned to perform the proper visuomotor transformations based on a context-modulated memory of the visual cue in its hidden layer activity. In network II, a brief visual response, which was driven by the sensory input, is quickly replaced by a context-modulated motor-goal representation in the hidden layer. This happens because of a dominant feedback signal from the output layer that first conveys context information, and then, after the disappearance of the visual cue, conveys motor goal information. We also show that the origin of the context information is not necessarily closely tied to the top-down feedback. However, we suggest that the predominance of motor-goal representations found in the parietal cortex during context-specific movement planning might be the consequence of strong top-down feedback originating from within the parietal lobe or from the frontal lobe.

Scale-invariance of receptive field properties in primary visual cortex

Background

Our visual system enables us to recognize visual objects across a wide range of spatial scales. The neural mechanisms underlying these abilities are still poorly understood. Size- or scale-independent representation of visual objects might be supported by processing in primary visual cortex (V1). Neurons in V1 are selective for spatial frequency and thus represent visual information in specific spatial wavebands. We tested whether different receptive field properties of neurons in V1 scale with preferred spatial wavelength. Specifically, we investigated the size of the area that enhances responses, i.e., the grating summation field, the size of the inhibitory surround, and the distance dependence of signal coupling, i.e., the linking field.

Results

We found that the sizes of both grating summation field and inhibitory surround increase with preferred spatial wavelength. For the summation field this increase, however, is not strictly linear. No evidence was found that size of the linking field depends on preferred spatial wavelength.

Conclusion

Our data show that some receptive field properties are related to preferred spatial wavelength. This speaks in favor of the hypothesis that processing in V1 supports scale-invariant aspects of visual performance. However, not all properties of receptive fields in V1 scale with preferred spatial wavelength. Spatial-wavelength independence of the linking field implies a constant spatial range of signal coupling between neurons with different preferred spatial wavelengths. This might be important for encoding extended broad-band visual features such as edges.

Neural dynamics in monkey parietal reach region reflect context-specific sensorimotor transformations

We investigated the neural dynamics of sensorimotor transformations in the parietal reach region (PRR) of monkeys. To dissociate sensory from motor goal representations, we used a memory-guided anti-reach task. The monkeys had to reach either to a visually instructed, memorized peripheral target position (pro-reach) or to a diametrically opposed position (anti) while keeping central ocular fixation. Pro- and anti-reaches were randomly interleaved and indicated by a color instruction from the beginning of each trial. We analyzed spatiotemporal single-cell tuning and performed time-resolved population decoding to quantify the dynamic representation of the spatial visual cue, the reach goal, and the currently valid task rule (pro/anti mapping). Sensory information regarding the visual cue position was represented weakly during a short period of cue visibility. PRR predominantly encoded the reach goal from the end of the cue period on. The representation of the reach goal in the memory task evolves later for the anti- compared with pro-reaches, consistent with a 40-50 ms difference in reaction time between the two task rules. The task rule could be decoded before the appearance of the spatial cue, which indicates that abstract rule information is present in PRR that is independent of spatial cue or motor goal representations. Our findings support the hypothesis that PRR immediately translates current sensory information into reach movement plans, rather than storing the memorized cue location in the instructed-delay task. This finding indicates that PRR represents integrated knowledge on spatial sensory information combined with abstract behavioral rules to encode the desired movement goal.

Complex dynamics of a single neuron model

We study a mathematical model of a single neuron with self-coupling. The model is based on the FitzHugh-Nagumo oscillator and an equation describing synaptic properties of the neuron. The analysis of the model is focused on its dynamics, depending on parameters characterizing synaptic time constants and external signals that affect the neuron. Applying Lyapunov exponents and bifurcation analysis, we point out the occurrence of parameter regions with different behavior such as bursting (chaotic or periodic), spiking, and multistable phenomena. Moreover, we can describe the dynamics of the model using an analytical approximation of the one-dimensional Poincaré map extracted from the numerical simulations.

Perception-related modulations of local field potential power and coherence in primary visual cortex of awake monkey during binocular rivalry

Cortical synchronization at gamma-frequencies (35-90 Hz) has been proposed to define the connectedness among the local parts of a perceived visual object. This hypothesis is still under debate. We tested it under conditions of binocular rivalry (BR), where a monkey perceived alternations among conflicting gratings presented singly to each eye at orthogonal orientations. We made multi-channel microelectrode recordings of multi-unit activity (MUA) and local field potentials (LFP) from striate cortex (V1) during BR while the monkey indicated his perception by pushing a lever. We analyzed spectral power and coherence of MUA and LFP over 4-90 Hz. As in previous work, coherence of gamma-signals in most pairs of recording locations strongly depended on grating orientation when stimuli were presented congruently in both eyes. With incongruent (rivalrous) stimulation LFP power was often consistently modulated in consonance with the perceptual state. This was not visible in MUA. These perception-related modulations of LFP occurred at low and medium frequencies (< 30 Hz), but not at gamma-frequencies. Perception-related modulations of LFP coherence were also restricted to the low-medium range. In conclusion, our results do not support the expectation that gamma-synchronization in V1 is related to the perceptual state during BR, but instead suggest a perception-related role of synchrony at low and medium frequencies.

Neural mechanisms of visual associative processing

This is a review of our work on multiple microelectrode recordings from the visual cortex of monkeys and subdural recordings from humans--related to the potential underlying neural mechanisms. The former hypothesis of object representation by synchronization in visual cortex (or more generally: of flexible associative processing) has been supported by our recent experiments in monkeys. They demonstrated local synchrony among rhythmic or stochastic gamma-activities (30-90 Hz) and perceptual modulation, according to the rules of figure-ground segregation. However, gamma-synchrony in primary visual cortex is restricted to few millimeters, challenging the synchronization hypothesis for larger cortical object representations. We found that the spatial restriction is due to gamma-waves, traveling in random directions across the object representations. It will be argued that phase continuity of these waves can support the coding of object continuity. Based on models with spiking neurons, potentially underlying neural mechanisms are proposed: (i) Fast inhibitory feedback loops can generate locally synchronized gamma-activities; (ii) Hebbian learning of lateral and feed forward connections with distance-dependent delays can explain the stabilization of cortical retinotopy, the limited size of synchronization, the occurrence of gamma-waves, and the larger receptive fields at successive levels; (iii) slow inhibitory feedback can support figure-ground segregation; (iv) temporal dispersion in far projections destroys coherence of fast signals but preserves slow amplitude modulations. In conclusion, it is proposed that the hypothesis of flexible associative processing by gamma-synchronization, including coherent representations of visual objects, has to be extended to more general forms of signal coupling.

Simultaneous mapping of binocular and monocular receptive fields in awake monkeys for calibrating eye alignment in a dichoptical setup

We developed a modified Wheatstone stereoscope for simultaneous dichoptical and binocular stimulation in awake monkey. We, therefore, extended the conventional two-screen Wheatstone stereoscope to a setup with an additional third screen viewed binocularly via semi-transparent mirrors. With a sparse noise stimulation we mapped classical receptive field (CRF) positions via each screen independently but simultaneously. This was done for multiple recording positions (16 electrodes) at once in primary visual cortex based on multiple unit spike activity (MUA) and local field potentials (LFP), respectively. The technique can be used to (1) quickly and simultaneously determine binocular as well as left and right eye CRFs, including ocular dominance characteristics (net recording time for the given examples: approximately 2 min), (2) precisely adjust dichoptical stimulation by evaluating offsets between monocular and binocular CRF positions (average spatial incongruency between binocular and left/right eye stimulation after calibration: approximately 0.025 degrees visual angle), and (3) investigate left and right eye interaction in forming binocular CRFs. Due to the precise adjustment of the dichoptical and the simultaneous binocular stimulation investigations on the basis of stereo vision can be done with appropriate eye vergence alignment matching normal binocular viewing conditions in awake animals.

Contour decouples gamma activity across texture representation in monkey striate cortex

Previous work on figure-ground coding in monkey V1 revealed enhanced spike rates within an object's surface representation, synchronization of gamma oscillations (gamma = 35-90 Hz) in object and background regions, but no decrease in signal correlation across the representation of a contour. The latter observation seems to contradict previous statements on the role of gamma-synchronization for scene segmentation. We re-examine these findings by analyzing different coupling measures and frequency ranges of population activities potentially contributing to figure-ground segregation. Multiple unit activity (MUA) and local field potentials (LFPs) were recorded by parallel mu-electrodes in monkey V1 during stimulation by a grating in which an object was defined by a shifted rectangle. In contradiction to the conclusions in previous work, we find strong decoupling of population activity between figure and ground representations compared to the situation in which the object is absent. In particular, coherence of late gamma-LFPs is strongly reduced, while reduction is absent during the early epochs of high-amplitude transients for LFP- and MUA-coherence at all frequencies, and at low frequencies also in the subsequent epochs. Our results of decoupling in late LFP gamma-components among figure and ground representations suggest that these signals may support figure-ground segregation.