Prediction of Reach Goals in Depth and Direction from the Parietal Cortex

Convolutional neural networks reveal properties of reach-to-grasp encoding in posterior parietal cortex

Deep neural networks (DNNs) are widely adopted to decode motor states from both non-invasively and invasively recorded neural signals, e.g., for realizing brain-computer interfaces. However, the neurophysiological interpretation of how DNNs make the decision based on the input neural activity is limitedly addressed, especially when applied to invasively recorded data. This reduces decoder reliability and transparency, and prevents the exploitation of decoders to better comprehend motor neural encoding. Here, we adopted an explainable artificial intelligence approach - based on a convolutional neural network and an explanation technique - to reveal spatial and temporal neural properties of reach-to-grasping from single-neuron recordings of the posterior parietal area V6A. The network was able to accurately decode 5 different grip types, and the explanation technique automatically identified the cells and temporal samples that most influenced the network prediction. Grip encoding in V6A neurons already started at movement preparation, peaking during movement execution. A difference was found within V6A: dorsal V6A neurons progressively encoded more for increasingly advanced grips, while ventral V6A neurons for increasingly rudimentary grips, with both subareas following a linear trend between the amount of grip encoding and the level of grip skills. By revealing the elements of the neural activity most relevant for each grip with no a priori assumptions, our approach supports and advances current knowledge about reach-to-grasp encoding in V6A, and it may represent a general tool able to investigate neural correlates of motor or cognitive tasks (e.g., attention and memory tasks) from single-neuron recordings.

The Arousal-Related "Central Thalamus" Stimulation Site Simultaneously Innervates Multiple High-Level Frontal and Parietal Areas

In human and nonhuman primates, deep brain stimulation applied at or near the internal medullary lamina of the thalamus [a region referred to as "central thalamus," (CT)], but not at nearby thalamic sites, elicits major changes in the level of consciousness, even in some minimally conscious brain-damaged patients. The mechanisms behind these effects remain mysterious, as the connections of CT had not been specifically mapped in primates. In marmoset monkeys (Callithrix jacchus) of both sexes, we labeled the axons originating from each of the various CT neuronal populations and analyzed their arborization patterns in the cerebral cortex and striatum. We report that, together, these CT populations innervate an array of high-level frontal, posterior parietal, and cingulate cortical areas. Some populations simultaneously target the frontal, parietal, and cingulate cortices, while others predominantly target the dorsal striatum. Our data indicate that CT stimulation can simultaneously engage a heterogeneous set of projection systems that, together, target the key nodes of the attention, executive control, and working-memory networks of the brain. Increased functional connectivity in these networks has been previously described as a signature of consciousness.SIGNIFICANCE STATEMENT In human and nonhuman primates, deep brain stimulation at a specific site near the internal medullary lamina of the thalamus ["central thalamus," (CT)] had been shown to restore arousal and awareness in anesthetized animals, as well as in some brain-damaged patients. The mechanisms behind these effects remain mysterious, as CT connections remain poorly defined in primates. In marmoset monkeys, we mapped with sensitive axon-labeling methods the pathways originated from CT. Our data indicate that stimulation applied in CT can simultaneously engage a heterogeneous set of projection systems that, together, target several key nodes of the attention, executive control, and working-memory networks of the brain. Increased functional connectivity in these networks has been previously described as a signature of consciousness.

Depth and direction effects in the prediction of static and shifted reaching goals from kinematics

The kinematic parameters of reach-to-grasp movements are modulated by action intentions. However, when an unexpected change in visual target goal during reaching execution occurs, it is still unknown whether the action intention changes with target goal modification and which is the temporal structure of the target goal prediction. We recorded the kinematics of the pointing finger and wrist during the execution of reaching movements in 23 naïve volunteers where the targets could be located at different directions and depths with respect to the body. During the movement execution, the targets could remain static for the entire duration of movement or shifted, with different timings, to another position. We performed temporal decoding of the final goals and of the intermediate trajectory from the past kinematics exploiting a recurrent neural network. We observed a progressive increase of the classification performance from the onset to the end of movement in both horizontal and sagittal dimensions, as well as in decoding shifted targets. The classification accuracy in decoding horizontal targets was higher than the classification accuracy of sagittal targets. These results are useful for establishing how human and artificial agents could take advantage from the observed kinematics to optimize their cooperation in three-dimensional space.

Early categorization of social affordances during the visual encoding of bodily stimuli

Interpersonal interactions rely on various communication channels, both verbal and non-verbal, through which information regarding one's intentions and emotions are perceived. Here, we investigated the neural correlates underlying the visual processing of hand postures conveying social affordances (i.e., hand-shaking), compared to control stimuli such as hands performing non-social actions (i.e., grasping) or showing no movement at all. Combining univariate and multivariate analysis on electroencephalography (EEG) data, our results indicate that occipito-temporal electrodes show early differential processing of stimuli conveying social information compared to non-social ones. First, the amplitude of the Early Posterior Negativity (EPN, an Event-Related Potential related to the perception of body parts) is modulated differently during the perception of social and non-social content carried by hands. Moreover, our multivariate classification analysis (MultiVariate Pattern Analysis - MVPA) expanded the univariate results by revealing early (<200 ms) categorization of social affordances over occipito-parietal sites. In conclusion, we provide new evidence suggesting that the encoding of socially relevant hand gestures is categorized in the early stages of visual processing.

Flexible intentions: An Active Inference theory

We present a normative computational theory of how the brain may support visually-guided goal-directed actions in dynamically changing environments. It extends the Active Inference theory of cortical processing according to which the brain maintains beliefs over the environmental state, and motor control signals try to fulfill the corresponding sensory predictions. We propose that the neural circuitry in the Posterior Parietal Cortex (PPC) compute flexible intentions-or motor plans from a belief over targets-to dynamically generate goal-directed actions, and we develop a computational formalization of this process. A proof-of-concept agent embodying visual and proprioceptive sensors and an actuated upper limb was tested on target-reaching tasks. The agent behaved correctly under various conditions, including static and dynamic targets, different sensory feedbacks, sensory precisions, intention gains, and movement policies; limit conditions were individuated, too. Active Inference driven by dynamic and flexible intentions can thus support goal-directed behavior in constantly changing environments, and the PPC might putatively host its core intention mechanism. More broadly, the study provides a normative computational basis for research on goal-directed behavior in end-to-end settings and further advances mechanistic theories of active biological systems.

Machine learning methods detect arm movement impairments in a patient with parieto-occipital lesion using only early kinematic information

Patients with lesions of the parieto-occipital cortex typically misreach visual targets that they correctly perceive (optic ataxia). Although optic ataxia was described more than 30 years ago, distinguishing this condition from physiological behavior using kinematic data is still far from being an achievement. Here, combining kinematic analysis with machine learning methods, we compared the reaching performance of a patient with bilateral occipitoparietal damage with that of 10 healthy controls. They performed visually guided reaches toward targets located at different depths and directions. Using the horizontal, sagittal, and vertical deviation of the trajectories, we extracted classification accuracy in discriminating the reaching performance of patient from that of controls. Specifically, accurate predictions of the patient's deviations were detected after the 20% of the movement execution in all the spatial positions tested. This classification based on initial trajectory decoding was possible for both directional and depth components of the movement, suggesting the possibility of applying this method to characterize pathological motor behavior in wider frameworks.

Decoding sensorimotor information from superior parietal lobule of macaque via Convolutional Neural Networks

Despite the well-recognized role of the posterior parietal cortex (PPC) in processing sensory information to guide action, the differential encoding properties of this dynamic processing, as operated by different PPC brain areas, are scarcely known. Within the monkey's PPC, the superior parietal lobule hosts areas V6A, PEc, and PE included in the dorso-medial visual stream that is specialized in planning and guiding reaching movements. Here, a Convolutional Neural Network (CNN) approach is used to investigate how the information is processed in these areas. We trained two macaque monkeys to perform a delayed reaching task towards 9 positions (distributed on 3 different depth and direction levels) in the 3D peripersonal space. The activity of single cells was recorded from V6A, PEc, PE and fed to convolutional neural networks that were designed and trained to exploit the temporal structure of neuronal activation patterns, to decode the target positions reached by the monkey. Bayesian Optimization was used to define the main CNN hyper-parameters. In addition to discrete positions in space, we used the same network architecture to decode plausible reaching trajectories. We found that data from the most caudal V6A and PEc areas outperformed PE area in the spatial position decoding. In all areas, decoding accuracies started to increase at the time the target to reach was instructed to the monkey, and reached a plateau at movement onset. The results support a dynamic encoding of the different phases and properties of the reaching movement differentially distributed over a network of interconnected areas. This study highlights the usefulness of neurons' firing rate decoding via CNNs to improve our understanding of how sensorimotor information is encoded in PPC to perform reaching movements. The obtained results may have implications in the perspective of novel neuroprosthetic devices based on the decoding of these rich signals for faithfully carrying out patient's intentions.

A Multifunctional Adaptive and Interactive AI system to support people living with stroke, acquired brain or spinal cord injuries: A study protocol

Background

Acquired brain injury and spinal cord injury are leading causes of severe motor disabilities impacting a person’s autonomy and social life. Enhancing neurological recovery driven by neurogenesis and neuronal plasticity could represent future solutions; however, at present, recovery of activities employing assistive technologies integrating artificial intelligence is worthy of examining. MAIA (Multifunctional, adaptive, and interactive AI system for Acting in multiple contexts) is a human-centered AI aiming to allow end-users to control assistive devices naturally and efficiently by using continuous bidirectional exchanges among multiple sensorimotor information.

Methods

Aimed at exploring the acceptability of MAIA, semi-structured interviews (both individual interviews and focus groups) are used to prompt possible end-users (both patients and caregivers) to express their opinions about expected functionalities, outfits, and the services that MAIA should embed, once developed, to fit end-users needs.

Discussion

End-user indications are expected to interest MAIA technical, health-related, and setting components. Moreover, psycho-social issues are expected to align with the technology acceptance model. In particular, they are likely to involve intrinsic motivational and extrinsic social aspects, aspects concerning the usefulness of the MAIA system, and the related ease to use. At last, we expect individual factors to impact MAIA: gender, fragility levels, psychological aspects involved in the mental representation of body image, personal endurance, and tolerance toward AT-related burden might be the aspects end-users rise in evaluating the MAIA project.

The interaction between motion and texture in the sense of touch

Besides providing information on elementary properties of objects, like texture, roughness, and softness, the sense of touch is also important in building a representation of object movement and the movement of our hands. Neural and behavioral studies shed light on the mechanisms and limits of our sense of touch in the perception of texture and motion, and of its role in the control of movement of our hands. The interplay between the geometrical and mechanical properties of the touched objects, such as shape and texture, the movement of the hand exploring the object, and the motion felt by touch, will be discussed in this article. Interestingly, the interaction between motion and textures can generate perceptual illusions in touch. For example, the orientation and the spacing of the texture elements on a static surface induces the illusion of surface motion when we move our hand on it or can elicit the perception of a curved trajectory during sliding, straight hand movements. In this work we present a multiperspective view that encompasses both the perceptual and the motor aspects, as well as the response of peripheral and central nerve structures, to analyze and better understand the complex mechanisms underpinning the tactile representation of texture and motion. Such a better understanding of the spatiotemporal features of the tactile stimulus can reveal novel transdisciplinary applications in neuroscience and haptics.

Neural encoding and functional interactions underlying pantomimed movements

Pantomimes are a unique movement category which can convey complex information about our intentions in the absence of any interaction with real objects. Indeed, we can pretend to use the same tool to perform different actions or to achieve the same goal adopting different tools. Nevertheless, how our brain implements pantomimed movements is still poorly understood. In our study, we explored the neural encoding and functional interactions underlying pantomimes adopting multivariate pattern analysis (MVPA) and connectivity analysis of fMRI data. Participants performed pantomimed movements, either grasp-to-move or grasp-to-use, as if they were interacting with two different tools (scissors or axe). These tools share the possibility to achieve the same goal. We adopted MVPA to investigate two levels of representation during the planning and execution of pantomimes: (1) distinguishing different actions performed with the same tool, (2) representing the same final goal irrespective of the adopted tool. We described widespread encoding of action information within regions of the so-called “tool” network. Several nodes of the network—comprising regions within the ventral and the dorsal stream—also represented goal information. The spatial distribution of goal information changed from planning—comprising posterior regions (i.e. parietal and temporal)—to execution—including also anterior regions (i.e. premotor cortex). Moreover, connectivity analysis provided evidence for task-specific bidirectional coupling between the ventral stream and parieto-frontal motor networks. Overall, we showed that pantomimes were characterized by specific patterns of action and goal encoding and by task-dependent cortical interactions.

Future Portrait of the Athletic Brain: Mechanistic Understanding of Human Sport Performance Via Animal Neurophysiology of Motor Behavior

Sport performances are often showcases of skilled motor control. Efforts to understand the neural processes subserving such movements may teach us about general principles of behavior, similarly to how studies on neurological patients have guided early work in cognitive neuroscience. While investigations on non-human animal models offer valuable information on the neural dynamics of skilled motor control that is still difficult to obtain from humans, sport sciences have paid relatively little attention to these mechanisms. Similarly, knowledge emerging from the study of sport performance could inspire innovative experiments in animal neurophysiology, but the latter has been only partially applied. Here, we advocate that fostering interactions between these two seemingly distant fields, i.e., animal neurophysiology and sport sciences, may lead to mutual benefits. For instance, recording and manipulating the activity from neurons of behaving animals offer a unique viewpoint on the computations for motor control, with potentially untapped relevance for motor skills development in athletes. To stimulate such transdisciplinary dialog, in the present article, we also discuss steps for the reverse translation of sport sciences findings to animal models and the evaluation of comparability between animal models of a given sport and athletes. In the final section of the article, we envision that some approaches developed for animal neurophysiology could translate to sport sciences anytime soon (e.g., advanced tracking methods) or in the future (e.g., novel brain stimulation techniques) and could be used to monitor and manipulate motor skills, with implications for human performance extending well beyond sport.

Mixed Selectivity in Macaque Medial Parietal Cortex during Eye-Hand Reaching

The activity of neurons of the medial posterior parietal area V6A in macaque monkeys is modulated by many aspects of reach task. In the past, research was mostly focused on modulating the effect of single parameters upon the activity of V6A cells. Here, we used Generalized Linear Models (GLMs) to simultaneously test the contribution of several factors upon V6A cells during a fix-to-reach task. This approach resulted in the definition of a representative “functional fingerprint” for each neuron. We first studied how the features are distributed in the population. Our analysis highlighted the virtual absence of units strictly selective for only one factor and revealed that most cells are characterized by “mixed selectivity.” Then, exploiting our GLM framework, we investigated the dynamics of spatial parameters encoded within V6A. We found that the tuning is not static, but changed along the trial, indicating the sequential occurrence of visuospatial transformations helpful to guide arm movement.

•

The parietal cortex integrates a variety of sensorimotor inputs to guide reaching

•

GLM disentangled the effect of various reaching parameters upon cell activity

•

V6A neurons were not functionally clustered, but characterized by mixed selectivity

•

Spatial selectivity was dynamic and reached its peak during the movement phase

Decoding of standard and non-standard visuomotor associations from parietal cortex

OBJECTIVE

Neural signals can be decoded and used to move neural prostheses with the purpose of restoring motor function in patients with mobility impairments. Such patients typically have intact eye movement control and visual function, suggesting that cortical visuospatial signals could be used to guide external devices. Neurons in parietal cortex mediate sensory-motor transformations, encode the spatial coordinates for reaching goals, hand position and movements, and other spatial variables. We studied how spatial information is represented at the population level, and the possibility to decode not only the position of visual targets and the plans to reach them, but also conditional, non-spatial motor responses.

APPROACH

The animals first fixated one of nine targets in 3D space and then, after the target changed color, either reached toward it, or performed a non-spatial motor response (lift hand from a button). Spiking activity of parietal neurons was recorded in monkeys during two tasks. We then decoded different task related parameters.

MAIN RESULTS

We first show that a maximum-likelihood estimation (MLE) algorithm trained separately in each task transformed neural activity into accurate metric predictions of target location. Furthermore, by combining MLE with a Naïve Bayes classifier, we decoded the monkey's motor intention (reach or hand lift) and the different phases of the tasks. These results show that, although V6A encodes the spatial location of a target during a delay period, the signals they carry are updated around the movement execution in an intention/motor specific way.

SIGNIFICANCE

These findings show the presence of multiple levels of information in parietal cortex that could be decoded and used in brain machine interfaces to control both goal-directed movements and more cognitive visuomotor associations.

Neural coding of action in three dimensions: Task- and time-invariant reference frames for visuospatial and motor-related activity in parietal area V6A

Goal-directed movements involve a series of neural computations that compare the sensory representations of goal location and effector position, and transform these into motor commands. Neurons in posterior parietal cortex (PPC) control several effectors (e.g., eye, hand, foot) and encode goal location in a variety of spatial coordinate systems, including those anchored to gaze direction, and to the positions of the head, shoulder, or hand. However, there is little evidence on whether reference frames depend also on the effector and/or type of motor response. We addressed this issue in macaque PPC area V6A, where previous reports using a fixate-to-reach in depth task, from different starting arm positions, indicated that most units use mixed body/hand-centered coordinates. Here, we applied singular value decomposition and gradient analyses to characterize the reference frames in V6A while the animals, instead of arm reaching, performed a nonspatial motor response (hand lift). We found that most neurons used mixed body/hand coordinates, instead of "pure" body-, or hand-centered coordinates. During the task progress the effect of hand position on activity became stronger compared to target location. Activity consistent with body-centered coding was present only in a subset of neurons active early in the task. Applying the same analyses to a population of V6A neurons recorded during the fixate-to-reach task yielded similar results. These findings suggest that V6A neurons use consistent reference frames between spatial and nonspatial motor responses, a functional property that may allow the integration of spatial awareness and movement control.

Brain-Inspired Coding of Robot Body Schema Through Visuo-Motor Integration of Touched Events

Representing objects in space is difficult because sensorimotor events are anchored in different reference frames, which can be either eye-, arm-, or target-centered. In the brain, Gain-Field (GF) neurons in the parietal cortex are involved in computing the necessary spatial transformations for aligning the tactile, visual and proprioceptive signals. In reaching tasks, these GF neurons exploit a mechanism based on multiplicative interaction for binding simultaneously touched events from the hand with visual and proprioception information.By doing so, they can infer new reference frames to represent dynamically the location of the body parts in the visual space (i.e., the body schema) and nearby targets (i.e., its peripersonal space). In this line, we propose a neural model based on GF neurons for integrating tactile events with arm postures and visual locations for constructing hand- and target-centered receptive fields in the visual space. In robotic experiments using an artificial skin, we show how our neural architecture reproduces the behaviors of parietal neurons (1) for encoding dynamically the body schema of our robotic arm without any visual tags on it and (2) for estimating the relative orientation and distance of targets to it. We demonstrate how tactile information facilitates the integration of visual and proprioceptive signals in order to construct the body space.

Prediction of Forelimb Reach Results From Motor Cortex Activities Based on Calcium Imaging and Deep Learning

Brain-wide activities revealed by neuroimaging and recording techniques have been used to predict motor and cognitive functions in both human and animal models. However, although studies have shown the existence of micrometer-scale spatial organization of neurons in the motor cortex relevant to motor control, two-photon microscopy (TPM) calcium imaging at cellular resolution has not been fully exploited for the same purpose. Here, we ask if calcium imaging data recorded by TPM in rodent brain can provide enough information to predict features of upcoming movement. We collected calcium imaging signal from rostral forelimb area in layer 2/3 of the motor cortex while mice performed a two-dimensional lever reaching task. Images of average calcium activity collected during motion preparation period and inter-trial interval (ITI) were used to predict the forelimb reach results. The evaluation was based on a deep learning model that had been applied for object recognition. We found that the prediction accuracy for both maximum reaching location and trial outcome based on motion preparation period but not ITI were higher than the probabilities governed by chance. Our study demonstrated that imaging data encompassing information on the spatial organization of functional neuronal clusters in the motor cortex is useful in predicting motor acts even in the absence of detailed dynamics of neural activities.

Mixed Spatial and Movement Representations in the Primate Posterior Parietal Cortex

The posterior parietal cortex (PPC) of humans and non-human primates plays a key role in the sensory and motor transformations required to guide motor actions to objects of interest in the environment. Despite decades of research, the anatomical and functional organization of this region is still a matter of contention. It is generally accepted that specialized parietal subregions and their functional counterparts in the frontal cortex participate in distinct segregated networks related to eye, arm and hand movements. However, experimental evidence obtained primarily from single neuron recording studies in non-human primates has demonstrated a rich mixing of signals processed by parietal neurons, calling into question ideas for a strict functional specialization. Here, we present a brief account of this line of research together with the basic trends in the anatomical connectivity patterns of the parietal subregions. We review, the evidence related to the functional communication between subregions of the PPC and describe progress towards using parietal neuron activity in neuroprosthetic applications. Recent literature suggests a role for the PPC not as a constellation of specialized functional subdomains, but as a dynamic network of sensorimotor loci that combine multiple signals and work in concert to guide motor behavior.

Reduced neural representation of arm/hand actions in the medial posterior parietal cortex

Several investigations at a single-cell level demonstrated that the medial posterior parietal area V6A is involved in encoding reaching and grasping actions in different visual conditions. Here, we looked for a “low-dimensional” representation of these encoding processes by studying macaque V6A neurons tested in three different tasks with a dimensionality reduction technique, the demixed principal component analysis (dPCA), which is very suitable for neuroprosthetics readout. We compared neural activity in reaching and grasping tasks by highlighting the portions of population variance involved in the encoding of visual information, target position, wrist orientation and grip type. The weight of visual information and task parameters in the encoding process was dependent on the task. We found that the distribution of variance captured by visual information in the three tasks did not differ significantly among the tasks, whereas the variance captured by target position and grip type parameters were significantly higher with respect to that captured by wrist orientation regardless of the number of conditions considered in each task. These results suggest a different use of relevant information according to the type of planned and executed action. This study shows a simplified picture of encoding that describes how V6A processes relevant information for action planning and execution.