Quantized Attention-Gated Kernel Reinforcement Learning for Brain-Machine Interface Decoding

Granger Causal Inference Based on Dual Laplacian Distribution and Its Application to MI-BCI Classification

Granger causality-based effective brain connectivity provides a powerful tool to probe the neural mechanism for information processing and the potential features for brain computer interfaces. However, in real applications, traditional Granger causality is prone to the influence of outliers, such as inevitable ocular artifacts, resulting in unreasonable brain linkages and the failure to decipher inherent cognition states. In this work, motivated by constructing the sparse causality brain networks under the strong physiological outlier noise conditions, we proposed a dual Laplacian Granger causality analysis (DLap-GCA) by imposing Laplacian distributions on both model parameters and residuals. In essence, the first Laplacian assumption on residuals will resist the influence of outliers in electroencephalogram (EEG) on causality inference, and the second Laplacian assumption on model parameters will sparsely characterize the intrinsic interactions among multiple brain regions. Through simulation study, we quantitatively verified its effectiveness in suppressing the influence of complex outliers, the stable capacity for model estimation, and sparse network inference. The application to motor-imagery (MI) EEG further reveals that our method can effectively capture the inherent hemispheric lateralization of MI tasks with sparse patterns even under strong noise conditions. The MI classification based on the network features derived from the proposed approach shows higher accuracy than other existing traditional approaches, which is attributed to the discriminative network structures being captured in a timely manner by DLap-GCA even under the single-trial online condition. Basically, these results consistently show its robustness to the influence of complex outliers and the capability of characterizing representative brain networks for cognition information processing, which has the potential to offer reliable network structures for both cognitive studies and future brain-computer interface (BCI) realization.

Revealing unexpected complex encoding but simple decoding mechanisms in motor cortex via separating behaviorally relevant neural signals

In motor cortex, behaviorally relevant neural responses are entangled with irrelevant signals, which complicates the study of encoding and decoding mechanisms. It remains unclear whether behaviorally irrelevant signals could conceal some critical truth. One solution is to accurately separate behaviorally relevant and irrelevant signals at both single-neuron and single-trial levels, but this approach remains elusive due to the unknown ground truth of behaviorally relevant signals. Therefore, we propose a framework to define, extract, and validate behaviorally relevant signals. Analyzing separated signals in three monkeys performing different reaching tasks, we found neural responses previously considered to contain little information actually encode rich behavioral information in complex nonlinear ways. These responses are critical for neuronal redundancy and reveal movement behaviors occupy a higher-dimensional neural space than previously expected. Surprisingly, when incorporating often-ignored neural dimensions, behaviorally relevant signals can be decoded linearly with comparable performance to nonlinear decoding, suggesting linear readout may be performed in motor cortex. Our findings prompt that separating behaviorally relevant signals may help uncover more hidden cortical mechanisms.

Use of Artificial Intelligence Techniques to Assist Individuals with Physical Disabilities

Assistive technologies (AT) enable people with disabilities to perform activities of daily living more independently, have greater access to community and healthcare services, and be more productive performing educational and/or employment tasks. Integrating artificial intelligence (AI) with various agents, including electronics, robotics, and software, has revolutionized AT, resulting in groundbreaking technologies such as mind-controlled exoskeletons, bionic limbs, intelligent wheelchairs, and smart home assistants. This article provides a review of various AI techniques that have helped those with physical disabilities, including brain-computer interfaces, computer vision, natural language processing, and human-computer interaction. The current challenges and future directions for AI-powered advanced technologies are also addressed.

Hierarchical Deep Reinforcement Learning-Based Propofol Infusion Assistant Framework in Anesthesia

This article aims to provide a hierarchical reinforcement learning (RL)-based solution to the automated drug infusion field. The learning policy is divided into the tasks of: 1) learning trajectory generative model and 2) planning policy model. The proposed deep infusion assistant policy gradient (DIAPG) model draws inspiration from adversarial autoencoders (AAEs) and learns latent representations of hypnotic depth trajectories. Given the trajectories drawn from the generative model, the planning policy infers a dose of propofol for stable sedation of a patient under total intravenous anesthesia (TIVA) using propofol and remifentanil. Through extensive evaluation, the DIAPG model can effectively stabilize bispectral index (BIS) and effect site concentration given a potentially time-varying target sequence. The proposed DIAPG shows an increased performance of 530% and 15% when a human expert and a standard reinforcement algorithm are used to infuse drugs, respectively.

Audio-induced medial prefrontal cortical dynamics enhances coadaptive learning in brain-machine interfaces

Objectives. Coadaptive brain-machine interfaces (BMIs) allow subjects and external devices to adapt to each other during the closed-loop control, which provides a promising solution for paralyzed individuals. Previous studies have focused on either improving sensory feedback to facilitate subject learning or developing adaptive algorithms to maintain stable decoder performance. In this work, we aim to design an efficient coadaptive BMI framework which not only facilitates the learning of subjects on new tasks with designed sensory feedback, but also improves decoders' learning ability by extracting sensory feedback-induced evaluation information.Approach. We designed dynamic audio feedback during the trial according to the subjects' performance when they were trained to learn a new behavioral task. We compared the learning performance of two groups of Sprague Dawley rats, one with and the other without the designed audio feedback to show whether this audio feedback could facilitate the subjects' learning. Compared with the traditional closed-loop in BMI systems, an additional closed-loop involving medial prefrontal cortex (mPFC) activity was introduced into the coadaptive framework. The neural dynamics of audio-induced mPFC activity was analyzed to investigate whether a significant neural response could be triggered. This audio-induced response was then translated into reward expectation information to guide the learning of decoders on a new task. The multiday decoding performance of the decoders with and without audio-induced reward expectation was compared to investigate whether the extracted information could accelerate decoders to learn a new task.Main results. The behavior performance comparison showed that the average days for rats to achieve 80% well-trained behavioral performance was improved by 26.4% after introducing the designed audio feedback sequence. The analysis of neural dynamics showed that a significant neural response of mPFC activity could be elicited by the audio feedback and the visualization of audio-induced neural patterns was emerged and accompanied by the behavioral improvement of subjects. The multiday decoding performance comparison showed that the decoder taking the reward expectation information could achieve faster task learning by 33.8% on average across subjects.Significance. This study demonstrates that the designed audio feedback could improve the learning of subjects and the mPFC activity induced by audio feedback can be utilized to improve the decoder's learning efficiency on new tasks. The coadaptive framework involving mPFC dynamics in the closed-loop interaction can advance the BMIs into a more adaptive and efficient system with learning ability on new tasks.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

Cluster Kernel Reinforcement Learning-based Kalman Filter for Three-Lever Discrimination Task in Brain-Machine Interface

Brain-Machine Interface (BMI) translates paralyzed people's neural activity into control commands of the prosthesis so that their lost motor functions could be restored. The neural activities represent brain states that change continuously over time which brings the challenge to the online decoder. Reinforcement Learning (RL) has the advantage to construct the dynamic neural-kinematic mapping during the interaction. However, existing RL decoders output discrete actions as a classification problem and cannot provide continuous estimation. Previous work has combined Kalman Filter (KF) with RL for BMI, which achieves a continuous motor state estimation. However, this method adopts a neural network structure, which might get stuck in local optimum and cannot provide an efficient online update for the neural-kinematic mapping. In this paper, we propose a Cluster Kernel Reinforcement Learning-based Kalman Filter (CKRL-based KF) to avoid the local optimum problem for online neural-kinematic updating. The neural patterns are projected into Reproducing Kernel Hilbert Space (RKHS), which builds a universal approximation to guarantee the global optimum. We compare our proposed algorithm with the existing method on rat data collected during a brain control three-lever discrimination task. Our preliminary results show that the proposed method has a higher trial accuracy with lower variance across data segments, which shows its potential to improve the performance for online BMI control. Clinical Relevance- This paper provides a more stable decoding method for adaptive and continuous neural decoding. It is promising for clinical applications in BMI.

Kernel Temporal Differences for EEG-based Reinforcement Learning Brain Machine Interfaces

Kernel temporal differences (KTD) (λ) algorithm integrated in Q-learning (Q-KTD) has shown its applicability and feasibility for reinforcement learning brain machine interfaces (RLBMIs). RLBMI with its unique learning strategy based on trial-error allows continuous learning and adaptation in BMIs. Q-KTD has shown good performance in both open and closed-loop experiments for finding a proper mapping from neural intention to control commands of an external device. However, previous studies have been limited to intracortical BMIs where monkey's firing rates from primary motor cortex were used as inputs to the neural decoder. This study provides the first attempt to investigate Q-KTD algorithm's applicability in EEG-based RLBMIs. Two different publicly available EEG data sets are considered, we refer to them as Data set A and Data set B. EEG motor imagery tasks are integrated in a single step center-out reaching task, and we observe the open-loop RLBMI experiments reach 100% average success rates after sufficient learning experience. Data set A converges after approximately 20 epochs for raw features and Data set B shows convergence after approximately 40 epochs for both raw and Fourier transform features. Although there still exist challenges to overcome in EEG-based RLBMI using Q-KTD, including increasing the learning speed, and optimization of a continuously growing number of kernel units, the results encourage further investigation of Q-KTD in closed-loop RLBMIs using EEG. Clinical Relevance- This study supports feasibility of noninvasive EEG-based RLBMI implementations and addresses benefits and challenges of RLBMI using EEG.

Kernel Temporal Difference based Reinforcement Learning for Brain Machine Interfaces. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021;2021:6721-6724. [PMID: 34892650 DOI: 10.1109/embc46164.2021.9631086] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Brain-machine interfaces (BMIs) enable people with disabilities to control external devices with their motor intentions through a decoder. Compared with supervised learning, reinforcement learning (RL) is more promising for the disabled because it can assist them to learn without actual limb movement. Current RL decoders deal with tasks with immediate reward delivery. But for tasks where the reward is only given by the end of the trial, existing RL methods may take a long time to train and are prone to becoming trapped in the local minima. In this paper, we propose to embed temporal difference method (TD) into Quantized Attention-Gated Kernel Reinforcement Learning (QAGKRL) to solve this temporal credit assignment problem. This algorithm utilizes a kernel network to ensure the global linear structure and adopts a softmax policy to efficiently explore the state-action mapping through TD error. We simulate a center-out task where the agent needs several steps to first reach a periphery target and then return to the center to get the external reward. Our proposed algorithm is tested on simulated data and compared with two state-of-the-art models. We find that introducing the TD method to QAGKRL achieves a prediction accuracy of 96.2% ± 0.77% (mean ± std), which is significantly better the other two methods.Clinical Relevance-This paper proposes a novel kernel temporal difference RL method for the multi-step task with delayed reward delivery, which potentially enables BMI online continuous decoding.

Robust neural decoding by kernel regression with Siamese representation learning

Objective. Brain-machine interfaces (BMIs) provide a direct pathway between the brain and external devices such as computer cursors and prosthetics, which have great potential in motor function restoration. One critical limitation of current BMI systems is the unstable performance, partly due to the variability of neural signals. Studies showed that neural activities exhibit trial-to-trial variability, and the preferred direction of neurons frequently changes under different conditions. Therefore, a fixed decoding function does not work well.Approach. To deal with the problems, we propose a novel kernel regression framework. The nonparametric kernel regression is used to fit diverse decoding functions by finding similar neural patterns to handle neural variations caused by varying tuning functions. Further, the representations of raw neural signals are learned by Siamese networks and constrained by kinematic parameters, which can alleviate neural variations caused by intrinsic noises and task-irrelevant information. The representations are jointly learned with the kernel regression framework in an end-to-end manner so that neural variations can be tackled effectively.Main results. Experiments on two datasets demonstrate that our approach outperforms most existing methods and significantly improves the robustness in challenging situations such as limited samples and missing channels.Significance. The proposed approach demonstrates robust performance with different conditions and provides a new and inspiring perspective toward robust BMI control.

Task Learning Over Multi-Day Recording via Internally Rewarded Reinforcement Learning Based Brain Machine Interfaces

Autonomous brain machine interfaces (BMIs) aim to enable paralyzed people to self-evaluate their movement intention to control external devices. Previous reinforcement learning (RL)-based decoders interpret the mapping between neural activity and movements using the external reward for well-trained subjects, and have not investigated the task learning procedure. The brain has developed a learning mechanism to identify the correct actions that lead to rewards in the new task. This internal guidance can be utilized to replace the external reference to advance BMIs as an autonomous system. In this study, we propose to build an internally rewarded reinforcement learning-based BMI framework using the multi-site recording to demonstrate the autonomous learning ability of the BMI decoder on the new task. We test the model on the neural data collected over multiple days while the rats were learning a new lever discrimination task. The primary motor cortex (M1) and medial prefrontal cortex (mPFC) spikes are interpreted by the proposed RL framework into the discrete lever press actions. The neural activity of the mPFC post the action duration is interpreted as the internal reward information, where a support vector machine is implemented to classify the reward vs. non-reward trials with a high accuracy of 87.5% across subjects. This internal reward is used to replace the external water reward to update the decoder, which is able to adapt to the nonstationary neural activity during subject learning. The multi-cortical recording allows us to take in more cortical recordings as input and uses internal critics to guide the decoder learning. Comparing with the classic decoder using M1 activity as the only input and external guidance, the proposed system with multi-cortical recordings shows a better decoding accuracy. More importantly, our internally rewarded decoder demonstrates the autonomous learning ability on the new task as the decoder successfully addresses the time-variant neural patterns while subjects are learning, and works asymptotically as the subjects' behavioral learning progresses. It reveals the potential of endowing BMIs with autonomous task learning ability in the RL framework.

An Efficient Group Recommendation Model With Multiattention-Based Neural Networks

Group recommendation research has recently received much attention in a recommender system community. Currently, several deep-learning-based methods are used in group recommendation to learn preferences of groups on items and predict the next ones in which groups may be interested. However, their recommendation effectiveness is disappointing. To address this challenge, this article proposes a novel model called a multiattention-based group recommendation model (MAGRM). It well utilizes multiattention-based deep neural network structures to achieve accurate group recommendation. We train its two closely related modules: vector representation for group features and preference learning for groups on items. The former is proposed to learn to accurately represent each group's deep semantic features. It integrates four aspects of subfeatures: group co-occurrence, group description, and external and internal social features. In particular, we employ multiattention networks to learn to capture internal social features for groups. The latter employs a neural attention mechanism to depict preference interactions between each group and its members and then combines group and item features to accurately learn group preferences on items. Through extensive experiments on two real-world databases, we show that MAGRM remarkably outperforms the state-of-the-art methods in solving a group recommendation problem.

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain-Machine Interface

BACKGROUND

For the nonstationarity of neural recordings in intracortical brain-machine interfaces, daily retraining in a supervised manner is always required to maintain the performance of the decoder. This problem can be improved by using a reinforcement learning (RL) based self-recalibrating decoder. However, quickly exploring new knowledge while maintaining a good performance remains a challenge in RL-based decoders.

METHODS

To solve this problem, we proposed an attention-gated RL-based algorithm combining transfer learning, mini-batch, and weight updating schemes to accelerate the weight updating and avoid over-fitting. The proposed algorithm was tested on intracortical neural data recorded from two monkeys to decode their reaching positions and grasping gestures.

RESULTS

The decoding results showed that our proposed algorithm achieved an approximate 20% increase in classification accuracy compared to that obtained by the non-retrained classifier and even achieved better classification accuracy than the daily retraining classifier. Moreover, compared with a conventional RL method, our algorithm improved the accuracy by approximately 10% and the online weight updating speed by approximately 70 times.

CONCLUSIONS

This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.

Binless Kernel Machine: Modeling Spike Train Transformation for Cognitive Neural Prostheses

Modeling spike train transformation among brain regions helps in designing a cognitive neural prosthesis that restores lost cognitive functions. Various methods analyze the nonlinear dynamic spike train transformation between two cortical areas with low computational eficiency. The application of a real-time neural prosthesis requires computational eficiency, performance stability, and better interpretation of the neural firing patterns that modulate target spike generation. We propose the binless kernel machine in the point-process framework to describe nonlinear dynamic spike train transformations. Our approach embeds the binless kernel to eficiently capture the feedforward dynamics of spike trains and maps the input spike timings into reproducing kernel Hilbert space (RKHS). An inhomogeneous Bernoulli process is designed to combine with a kernel logistic regression that operates on the binless kernel to generate an output spike train as a point process. Weights of the proposed model are estimated by maximizing the log likelihood of output spike trains in RKHS, which allows a global-optimal solution. To reduce computational complexity, we design a streaming-based clustering algorithm to extract typical and important spike train features. The cluster centers and their weights enable the visualization of the important input spike train patterns that motivate or inhibit output neuron firing. We test the proposed model on both synthetic data and real spike train data recorded from the dorsal premotor cortex and the primary motor cortex of a monkey performing a center-out task. Performances are evaluated by discrete-time rescaling Kolmogorov-Smirnov tests. Our model outperforms the existing methods with higher stability regardless of weight initialization and demonstrates higher eficiency in analyzing neural patterns from spike timing with less historical input (50%). Meanwhile, the typical spike train patterns selected according to weights are validated to encode output spike from the spike train of single-input neuron and the interaction of two input neurons.

Self-adaptive shared control with brain state evaluation network for human-wheelchair cooperation

OBJECTIVE

For the shared control systems, how to trade off the control weight between robot autonomy and human operator is an important issue, especially for BCI-based systems. However, most of existing shared controllers have paid less attention to the effects caused by subjects with different levels of brain control ability.

APPROACH

In this paper, a brain state evaluation network, termed BSE-NET, is proposed to evaluate subjects' brain control ability online based on quantized attention-gated kernel reinforcement learning. With the output of BSE-NET (confidence score), a shared controller is designed to dynamically adjust the control weight between robot autonomy and human operator.

MAIN RESULTS

The experimental results show that most of subjects achieved high and stable experimental success rate of approximately 90%. Furthermore, for subjects with different accuracy on EEG decoding, a proper confidence score can be dynamically generated to reflect their levels of brain control ability, and the proposed system can effectively adjust the control weight in all-time shared control.

SIGNIFICANCE

We discuss how our proposed method shows promise for BCI applications that can evaluate subjects' brain control ability online as well as provide a method for the research on self-adaptive shared control to adaptively balance control weight between subject's instruction and robot autonomy.

Covariant Cluster Transfer for Kernel Reinforcement Learning in Brain-Machine Interface

Brain-Machine Interface (BMI) provides a promising way to help disabled people restore their motor functions. The patients are able to control the external devices directly from their neural signals by the decoder. Due to various reasons such as mental fatigue and distraction, the distribution of the neural signals might change, which might lead to poor performance for the decoder. In this case, we need to calibrate the parameters before each session, which needs the professionals to label the data and is not convenient for the patient's usage at home. In this paper, we propose a covariant cluster transfer mechanism for the kernel reinforcement learning (RL) algorithm to speed up the adaptation across sessions. The parameters of the decoder will adaptively change according to a reward signal, which could be easily set by the patient. More importantly, we cluster the neural patterns in previous sessions. The cluster represents the conditional distribution from neural patterns to actions. When a distinct neural pattern appears in the new session, the nearest cluster will be transferred. In this way, the knowledge from the old session could be utilized to accelerate the learning in the new session. Our proposed algorithm is tested on the simulated neural data where the neural signal's distribution differs across sessions. Compared with the training from random initialization and a weight transfer policy, our proposed cluster transfer mechanism maintains a significantly higher success rate and a faster adaptation when the conditional distribution from neural signals to actions remains similar.

Adaptive Neural Quantized Control for a Class of MIMO Switched Nonlinear Systems With Asymmetric Actuator Dead-Zone

This paper concentrates on the adaptive state-feedback quantized control problem for a class of multiple-input-multiple-output (MIMO) switched nonlinear systems with unknown asymmetric actuator dead-zone. In this study, we employ different quantizers for different subsystem inputs. The main challenge of this study is to deal with the coupling between the quantizers and the dead-zone nonlinearities. To solve this problem, a novel approximation model for the coupling between quantizer and dead-zone is proposed. Then, the corresponding robust adaptive law is designed to eliminate this nonlinear term asymptotically. A direct neural control scheme is employed to reduce the number of adaptive laws significantly. The backstepping-based adaptive control scheme is also presented to guarantee the system performance. Finally, two simulation examples are presented to show the effectiveness of our control scheme.

A Weight Transfer Mechanism for Kernel Reinforcement Learning Decoding in Brain-Machine Interfaces

Brain-Machine Interfaces (BMIs) aim to help disabled people brain control the external devices to finish a variety of movement tasks. The neural signals are decoded into the execution commands of the apparatus. However, most of the existing decoding algorithms in BMI are only trained for a single task. When facing a new task, even if it is similar to the previous one, the decoder needs to be re-trained from scratch, which is not efficient. Among the different types of decoders, reinforcement learning (RL) based algorithm has the advantage of adaptive training through trial-and-error over the recalibration used in supervised learning. But most of the RL algorithms in BMI do not actively leverage the acquired knowledge in the old task. In this paper, we propose a kernel RL algorithm with a weight transfer mechanism for new task learning. The existing neural patterns are clustered according to their similarities. A new pattern will be assigned with the weights that are transferred from the closest cluster. In this way, the most similar experiences from the previous task could be re-utilized in the new task to fasten the learning speed. The proposed algorithm is tested on synthetic neural data. Compared with the policy of re-training from scratch, the proposed weight transfer mechanism could maintain a significantly higher performance and achieve a faster learning speed on the new task.

Clustering Neural Patterns in Kernel Reinforcement Learning Assists Fast Brain Control in Brain-Machine Interfaces

Neuroprosthesis enables the brain control on the external devices purely using neural activity for paralyzed people. Supervised learning decoders recalibrate or re-fit the discrepancy between the desired target and decoder's output, where the correction may over-dominate the user's intention. Reinforcement learning decoder allows users to actively adjust their brain patterns through trial and error, which better represents the subject's motive. The computational challenge is to quickly establish new state-action mapping before the subject becomes frustrated. Recently proposed quantized attention-gated kernel reinforcement learning (QAGKRL) explores the optimal nonlinear neural-action mapping in the Reproducing Kernel Hilbert Space (RKHS). However, considering all past data in RKHS is less efficient and sensitive to detect the new neural patterns emerging in brain control. In this paper, we propose a clustering-based kernel RL algorithm. New neural patterns emerge and are clustered to represent the novel knowledge in brain control. The current neural data only activate the nearest subspace in RKHS for more efficient decoding. The dynamic clustering makes our algorithm more sensitive to new brain patterns. We test our algorithm on both the synthetic and real-world spike data. Compared with QAGKRL, our algorithm can achieve a quicker knowledge adaptation in brain control with less computational complexity.

Clustering Based Kernel Reinforcement Learning for Neural Adaptation in Brain-Machine Interfaces

Reinforcement learning (RL) interprets subject's movement intention in Brain Machine Interfaces (BMIs) through trial-and-error with the advantage that it does not need the real limb movements. When the subjects try to control the external devices purely using brain signals without actual movements (brain control), they adjust the neural firing patterns to adapt to device control, which expands the state-action space for the RL decoder to explore. The challenge is to quickly explore the new knowledge in the sizeable state-action space and maintain good performance. Recently quantized attention-gated kernel reinforcement learning (QAGKRL) was proposed to quickly explore the global optimum in Reproducing Kernel Hilbert Space (RKHS). However, its network size will grow large when the new input comes, which makes it computationally inefficient. In addition, the output is generated using the whole input structure without being sensitive to the new knowledge. In this paper, we propose a new kernel based reinforcement learning algorithm that utilizes the clustering technique in the input domain. The similar neural inputs are grouped, and a new input only activates its nearest cluster, which either utilizes an existing sub-network or forms a new one. In this way, we can build the sub-feature space instead of the global mapping to calculate the output, which transfers the old knowledge effectively and also consequently reduces the computational complexity. To evaluate our algorithm, we test on the synthetic spike data, where the subject's task mode switches between manual control and brain control. Compared with QAGKRL, the simulation results show that our algorithm can achieve a faster learning curve, less computational time, and more accuracy. This indicates our algorithm to be a promising method for the online implementation of BMIs.

Applications of Deep Learning and Reinforcement Learning to Biological Data

Rapid advances in hardware-based technologies during the past decades have opened up new possibilities for life scientists to gather multimodal data in various application domains, such as omics, bioimaging, medical imaging, and (brain/body)-machine interfaces. These have generated novel opportunities for development of dedicated data-intensive machine learning techniques. In particular, recent research in deep learning (DL), reinforcement learning (RL), and their combination (deep RL) promise to revolutionize the future of artificial intelligence. The growth in computational power accompanied by faster and increased data storage, and declining computing costs have already allowed scientists in various fields to apply these techniques on data sets that were previously intractable owing to their size and complexity. This paper provides a comprehensive survey on the application of DL, RL, and deep RL techniques in mining biological data. In addition, we compare the performances of DL techniques when applied to different data sets across various application domains. Finally, we outline open issues in this challenging research area and discuss future development perspectives.