Combining Brain-Computer Interfaces and Assistive Technologies: State-of-the-Art and Challenges

In recent years, new research has brought the field of electroencephalogram (EEG)-based brain–computer interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely, “Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user–machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human–computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices.

Estimation of the cortical activity from simultaneous multi-subject recordings during the prisoner's dilemma

One of the most challenging questions open in Neuroscience today is the characterization of the brain responses during social interaction. A major limitation of the approaches used in most of the studies performed so far is that only one of the participating brains is measured each time. The "interaction" between cooperating, competing or communicating brains is thus not measured directly, but inferred by independent observations aggregated by cognitive models and assumptions that link behavior and neural activation. In this paper, we present the results of the simultaneous neuroelectric recording of 5 couples of subjects engaged in cooperative games (EEG hyperscanning). The simultaneous recordings of couples of interacting subjects allows to observe and model directly the neural signature of human interactions in order to understand the cerebral processes generating and generated by social cooperation or competition. We used a paradigm called Prisoner's dilemma derived from the game theory. Results collected in a population of 10 subjects suggested that the most consistently activated structure in social interaction paradigms is the orbitofrontal region (roughly described by the Brodmann area 10) during the condition of competition.

Study of the time-varying cortical connectivity changes during the attempt of foot movements by spinal cord injured and healthy subjects

In this study we estimated time-varying cortical connectivity patterns from a group of Spinal Cord Injured (SCI) patients during the attempt to move a paralyzed limb. These data were compared with the time-varying connectivity patterns estimated in a control group during the real execution of the movement by using time-varying Partial Directed Coherence. Connectivity was estimated from high resolution EEG recordings with the use of realistic head modelling and the linear inverse estimation of the cortical activity in a series of Regions of Interest of the cortex (ROIs). The experimental evidences obtained support the conclusion that the SCI population involved a larger cortical network than those generated by the healthy subjects during the task performance. Such network differs for the involvement of the parietal cortices, which increases in strength near to the movement imagination onset for the SCI when compared to the normal population. Such details about the temporal evolution of the connectivity patterns cannot be obtained with the application of the standard estimators of connectivity.

Analysis of the connection redundancy in functional networks from high-resolution EEG: a preliminary study

In the present study, we propose a theoretical graph procedure to investigate the communication redundancy in brain networks. By taking into account all the possible paths between pairs of cortical regions, this method captures the network redundancy i.e. a critical resource of the brain enhancing the resilience to neural damages and dysfunctions. As an example for its potential, we apply this procedure to the cortical networks estimated from high-resolution EEG signals in a group of spinal cord injured patients during the attempt of the foot movement. Preliminary results suggest that in the high spectral contents the effects due to the spinal trauma affect the expected redundancy attitude by suppressing mainly the longer alternative pathways between the cortical regions.

Assessing the memorization of TV commercials with the use of high resolution EEG: a pilot study

The present work intends to evaluate the functional characteristics of the cerebral network during the successful memory encoding of TV commercials. We estimated the functional networks in the frequency domain from a set of high-resolution EEG data. High resolution EEG recordings were performed in a group of healthy subjects and the cortical activity during the observation of TV commercials was evaluated in several regions of interest coincident with the Brodmann areas (BAs). Summarizing the main results of the present study, a sign of the memorization of a particular set of TV commercials have been found in a group of investigated subjects with the aid of advanced modern tools for the acquisition and the processing of EEG data. The cerebral processes involved during the observation of TV commercials that were remembered successively by the population examined (RMB dataset) are generated by the posterior parietal cortices and the prefrontal areas, rather bilaterally and are irrespective of the frequency bands analyzed. Such results are compatible with previously results obtained from EEG recordings with superficial electrodes as well as with the brain activations observed with the use of MEG and fMRI devices.

Community structure of cortical networks in spinal cord injured patients

In the present study, we estimated the cortical networks were from high-resolution EEG recordings in a group of spinal cord injured patients and in a group of healthy subjects, during the preparation of a limb movement. Then, we use the Markov Clustering method to analyse the division of the network into community structures. The results indicate large differences between the injured patients and the healthy subjects. In particular, the networks of spinal cord injured patient exhibited a higher density of clusters. In the Alpha (7-12 Hz) frequency band, the two observed largest communities were mainly composed by the cingulate motor areas with the supplementary motor areas, and by the pre-motor areas with the right primary motor area of the foot. This functional separation could reflect the partial alteration in the primary motor areas because of the effects of the spinal cord injury.

Brain network analysis from high-resolution EEG recordings by the application of theoretical graph indexes

The extraction of the salient characteristics from brain connectivity patterns is an open challenging topic since often the estimated cerebral networks have a relative large size and complex structure. Since a graph is a mathematical representation of a network, which is essentially reduced to nodes and connections between them, the use of a theoretical graph approach would extract significant information from the functional brain networks estimated through different neuroimaging techniques. The present work intends to support the development of the "brain network analysis:" a mathematical tool consisting in a body of indexes based on the graph theory able to improve the comprehension of the complex interactions within the brain. In the present work, we applied for demonstrative purpose some graph indexes to the time-varying networks estimated from a set of high-resolution EEG data in a group of healthy subjects during the performance of a motor task. The comparison with a random benchmark allowed extracting the significant properties of the estimated networks in the representative Alpha (7-12 Hz) band. Altogether, our findings aim at proving how the brain network analysis could reveal important information about the time-frequency dynamics of the functional cortical networks.

Study of the time-varying cortical connectivity during the attempt of a foot movement by Spinal Cord Injured patients

In this study we estimated time-varying cortical connectivity patterns from a group of Spinal Cord Injured (SCI) patients during the attempt to move a paralyzed limb. This data were compared with the time-varying connectivity patterns estimated in a control group during the effective execution of the movement. Connectivity was estimated from high resolution EEG recordings with the use of realistic head modelling and the linear inverse estimation of the cortical activity. Time-varying PDC was obtained by the adaptive recursive fit of an MVAR model with time-dependent parameters, by means of a generalized recursive least-square (RLS) algorithm, taking into consideration a set of EEG epochs. Such estimator is able to follow rapid changes in the connectivity between cortical areas during an experimental task. The obtained experimental evidences support the conclusion that the SCI population involved a larger cortical network than those generated by the healthy subjects during the task performance. Such network differs for the involvement of the parietal cortices, which increases in strength near to the EMG onset.

Estimation of effective and functional cortical connectivity from neuroelectric and hemodynamic recordings

In this paper, different linear and nonlinear methodologies for the estimation of cortical connectivity from neuroelectric and hemodynamic measurements are reviewed and applied on common data set in order to highlight similarities and differences in the results. Different effective and functional connectivity methods were applied to motor and cognitive data sets, including structural equation modeling (SEM), directed transfer function (DTF), partial directed coherence (PDC), and direct directed transfer function (dDTF). Comparisons were made between the results in order to understand if, for a same dataset, effective and functional connectivity estimators can return the same cortical connectivity patterns. An application of a nonlinear method [phase synchronization index (PSI)] to similar executed and imagined movements was also reviewed. Connectivity patterns estimated with the use of the neuroelectric information and of the information from the multimodal integration of neuroelectric and hemodynamic data were also compared. Results suggests that the estimation of the cortical connectivity patterns performed with the linear methods (SEM, DTF, PDC, dDTF) or with the nonlinear method (PSI) on movement related potentials returned similar cortical networks. Differences in cortical connectivity were noted between the patterns estimated with the use of multimodal integration and those estimated by using only the neuroelectric data.

Cortical activity and connectivity of human brain during the prisoner's dilemma: an EEG hyperscanning study

A major limitation of the approaches used in most of the studies performed so far for the characterization of the brain responses during social interaction is that only one of the participating brains is measured each time. The "interaction" between cooperating, competing or communicating brains is thus not measured directly, but inferred by independent observations aggregated by cognitive models and assumptions that link behavior and neural activation. In this paper, we use the simultaneous neuroelectric recording of several subjects engaged in cooperative games (EEG hyperscanning). This EEG hyperscanning allow us to observe and model directly the neural signature of human interactions in order to understand the cerebral processes generating and generated by social cooperation or competition. We used a paradigm called Prisoner's dilemma derived from the game theory. Results collected in a population of 22 subjects suggested that the most consistently activated structure in social interaction paradigms is the medial prefrontal cortex, which is found to be active in all the conflict situations analyzed. The role of the anterior cingulated cortex (ACC) assumes a main character being a discriminant factor for the "defect" attitude of the entire population examined. This observation is compatible with the role that the Theory of Mind assigns to the ACC.

High resolution EEG hyperscanning during a card game

In order to study the concurrent activity in subjects interacting in cooperation or competition activities, the issue of the simultaneous recording of their brain activity became mandatory. The simultaneous recording of neuroelectric activity of the brain is called "EEG hyperscanning". We would like present results obtained by EEG hyperscannings performed on a group of subjects engaged in a card game. The EEG hyperscannings have been performed with the simultaneous use of high resolution EEG devices on groups of four subjects while they were playing a card game. We estimated the concurrent activity in multiple brains of the group and we depicted the causal connections between regions of different brains. Results obtained in a study of several groups recorded by the EEG hyperscanning reveal larger activity in prefrontal and anterior cingulated cortex in different frequency bands for the player that start the game when compared to other players. EEG hyperscannings will open a different area for the study of neuroscience, in which the activity of multiple brains during social cooperation could be investigated.

Tracking the time-varying cortical connectivity patterns by adaptive multivariate estimators

The directed transfer function (DTF) and the partial directed coherence (PDC) are frequency-domain estimators that are able to describe interactions between cortical areas in terms of the concept of Granger causality. However, the classical estimation of these methods is based on the multivariate autoregressive modelling (MVAR) of time series, which requires the stationarity of the signals. In this way, transient pathways of information transfer remains hidden. The objective of this study is to test a time-varying multivariate method for the estimation of rapidly changing connectivity relationships between cortical areas of the human brain, based on DTF/PDC and on the use of adaptive MVAR modelling (AMVAR) and to apply it to a set of real high resolution EEG data. This approach will allow the observation of rapidly changing influences between the cortical areas during the execution of a task. The simulation results indicated that time-varying DTF and PDC are able to estimate correctly the imposed connectivity patterns under reasonable operative conditions of signal-to-noise ratio (SNR) ad number of trials. An SNR of five and a number of trials of at least 20 provide a good accuracy in the estimation. After testing the method by the simulation study, we provide an application to the cortical estimations obtained from high resolution EEG data recorded from a group of healthy subject during a combined foot-lips movement and present the time-varying connectivity patterns resulting from the application of both DTF and PDC. Two different cortical networks were detected with the proposed methods, one constant across the task and the other evolving during the preparation of the joint movement.

Time-varying cortical connectivity by adaptive multivariate estimators applied to a combined foot-lips movement

In this paper we propose the use of an adaptive multivariate approach to define time-varying multivariate estimators based on the Directed Transfer Function (DTF) and the Partial Directed Coherence (PDC). DTF and PDC are frequency-domain estimators that are able to describe interactions between cortical areas in terms of the concept of Granger causality. Time-varying DTF and PDC were obtained by the adaptive recursive fit of an MVAR model with time-dependent parameters, by means of a generalized recursive least-square (RLS) algorithm, taking into consideration a set of EEG epochs. Such estimators are able to follow rapid changes in the connectivity between cortical areas during an experimental task. We provide an application to the cortical estimations obtained from high resolution EEG data, recorded from a group of healthy subject during a combined foot-lips movement, and present the time-varying connectivity patterns resulting from the application of both DTF and PDC. Two different cortical networks were detected, one constant across the task and the other evolving during the preparation of the joint movement.

Estimation of the time-varying cortical connectivity patterns by the adaptive multivariate estimators in high resolution EEG studies

The Directed Transfer Function (DTF) and the Partial Directed Coherence (PDC) are frequency-domain estimators, based on the multivariate autoregressive modelling (MVAR) of time series, that are able to describe interactions between cortical areas in terms of the concept of Granger causality. However, the classical estimation of these methods requires the stationary of the signals. In this way, transient pathways of information transfer remains hidden. The objective of this study is to test a time-varying multivariate method for the estimation of rapidly changing connectivity relationships between cortical areas of the human brain, based on DTF/PDC and on the use of adaptive MVAR modelling (AMVAR). This approach will allow the observation of transient influences between the cortical areas during the execution of a task. Time-varying DTF and PDC were obtained by the adaptive recursive fit of an MVAR model with time-dependent parameters, by means of a generalized recursive least-square (RLS) algorithm, taking into consideration a set of EEG epochs. Simulations were performed under different levels of Signal to Noise Ratio (SNR), number of trials (TRIALS) and frequency bands (BAND), and of different values of the RLS adaptation factor adopted (factor C). The results indicated that time-varying DTF and PDC are able to estimate correctly the imposed connectivity patterns under reasonable operative conditions of SNR ad number of trials. Moreover, the capability of follow the rapid changes in connectivity is highly increased by the number of trials at disposal, and by the right choice of the value adopted for the adaptation factor C. The results of the simulation study indicate that DTF and PDC computed on adaptive MVAR can be effectively used to estimate time-varying patterns of functional connectivity between cortical activations, under general conditions met in practical EEG recordings.

Cortical network topology during successful memory encoding in a lifelike experiment

In the present work, we estimated the functional networks in the frequency domain from a set of high-resolution EEG data in a group of healthy subjects during the showing of commercial spots within a neutral documentary. Then, we evaluated the differences in the cortical network associated with later remembered and not-remembered commercials by calculating the global- E(g) and local-efficiency E(l) indexes. During the visualization of the video-clips that will be forgotten (FRG), the cortical network exhibited high values of global- and local-efficiency, reflecting a small-world configuration. During the visualization of the video-clips that will be remembered (RMB), the same indexes appeared significantly lower. Such a difference seems not depending on the spectral content of the cortical activity. This result shows how the network communication efficiency would be affected by the presence of attentional and semantic processes that are behind a successful memory encoding in a lifelike situation.

Imaging functional brain connectivity patterns from high-resolution EEG and fMRI via graph theory

We describe a set of computational tools able to estimate cortical activity and connectivity from high-resolution EEG and fMRI recordings in humans. These methods comprise the estimation of cortical activity using realistic geometry head volume conductor models and distributed cortical source models, followed by the evaluation of cortical connectivity between regions of interest coincident with the Brodmann areas via the use of Partial Directed Coherence. Connectivity patterns estimated on the cortical surface in different frequency bands are then imaged and interpreted with measures based on graph theory. These computational tools were applied on a set of EEG and fMRI data from a Stroop task to demonstrate the potential of the proposed approach. The present findings suggest that the methodology is able to identify differences in functional connectivity patterns elicited by different experimental tasks or conditions.

Neuroelectrical source imaging of mu rhythm control for BCI applications

In the last decade, the possibility to noninvasively estimate cortical activity has been highlighted by the application of the techniques known as high resolution EEG. These techniques include a subject's multi-compartment head model (scalp, skull, dura mater, cortex) constructed from individual magnetic resonance images, multi-dipole source model, and regularized linear inverse source estimates of cortical current density. The aim of this paper is to demonstrate that the use of cortical activity estimated from noninvasive EEG recordings of motor imagery is useful in the context of a brain computer interface as compared with others scalp spatial filters usually used on-line.

Brain computer interface: estimation of cortical activity from non invasive high resolution EEG recordings

The aim of this paper is to analyze whether the use of the cortical activity estimated from non invasive EEG recordings could be useful to detect mental states related to the imagination of limb movements. Estimation of cortical activity was performed on high resolution EEG data related to the imagination of limb movements gathered in five normal healthy subjects by using realistic head models. Cortical activity was estimated in region of interest associated with the subject's Brodmann areas by using depth-weighted minimum norm solutions. Comparisons between surface recorded EEG and the estimated cortical activity were performed. The estimated cortical activity related to the mental imagery of limbs in the five subjects is located mainly over the contralateral primary motor area. The unbalance between brain activity estimated in contralateral and ipsilateral motor cortical areas relative to the finger movement imagination is greater than those obtained in the scalp EEG recordings. Results suggest that the use of the estimated cortical activity for the motor imagery of upper limbs could be potentially superior with respect to the use of surface EEG recordings. This is due to a greater statistically significant unbalance between the activity estimated in the contralateral and ipsilateral hemisphere with respect to those observed with surface EEG. These results are useful in the context of the development of a non invasive brain computer interface.

Improved estimation of human cortical activity and connectivity with the multimodal integration of neuroelectric and hemodynamic data

In the last decade, the possibility to noninvasively estimate cortical activity and connectivity has been highlighted by the application of the techniques known as high resolution EEG. These techniques include a subject's multi-compartment head model (scalp, skull, dura mater, cortex) constructed from individual magnetic resonance images, multi-dipole source model, and regularized linear inverse source estimates of cortical current density. More recently, it has proved as the use of information from the hemodynamic responses of the cortical areas as revealed by block-designed (strength of activated voxels) fMRI improves dramatically the estimates of cortical activity and connectivity. Here, we present some applications of such estimation in two set of high resolution EEG and fMRI data, related to the motor (finger tapping) and cognitive (Stroop) tasks. We observed that the proposed technology was able to unveil the direction of the information flow between the cortical regions of interest.

Time-varying cortical connectivity by high resolution EEG and directed transfer function: simulations and application to finger tapping data

The problem of the definition and evaluation of brain connectivity has become a central one in neuroscience during the latest years, as a way to understand the organization and interaction of cortical areas during the execution of cognitive or motor tasks. The method of the directed transfer function (DTF) is a frequency-domain approach to this problem, based on a multivariate autoregressive modeling of time series and on the concept of Granger causality. So far, all the connectivity estimations performed on cerebral electromagnetic signals were computed between signals gathered from the electric or magnetic sensors. However, the spreading of the potential from the cortex to the sensors makes it difficult to infer the relation between the spatial patterns on the sensor space and those on the cortical sites. In this paper we propose the use of the DTF method on cortical signals estimated from high resolution EEG recordings, which exhibit a higher spatial resolution than conventional cerebral electromagnetic measures. As main contributions of this work, we present the results of a wide simulation study, aiming to evaluate performances of DTF application on this kind of data, and a statistical analysis (via the ANOVA, analysis of variance) of the results obtained for different levels of signal to noise ratio and temporal length, as they have been systematically imposed on simulated signals. Finally, we provide an application to the estimation of cortical connectivity from high resolution EEG recordings related to finger tapping movements.

Features extraction from time-varying cortical networks adopting a theoretical graph approach

In this work, a novel approach is proposed in order to capture relevant features related to the structure and organization of the functional brain networks estimated in the time-frequency domain. To achieve this, we used a cascade of computational tools able to estimate first the electrical activity of the cortical surface by using high resolution EEG techniques. Then, on the cortical signals from different regions of interests we estimated the time-varying functional connectivity patterns by means of the adaptive Partial Directed Coherence. Such time-varying connectivity estimation returns a series of causality patterns evolving during the examined task which can be summarized and interpreted with the aid of mathematical indexes based on the graph theory. The combination of all these methods is demonstrated on a set of high resolution EEG data recorded from a healthy subject performing a simple foot movement.