A Weight Transfer Mechanism for Kernel Reinforcement Learning Decoding in Brain-Machine Interfaces

Kernel Reinforcement Learning-Assisted Adaptive Decoder Facilitates Stable and Continuous Brain Control Tasks

Brain-Machine Interfaces (BMIs) assist paralyzed people to brain control (BC) the neuro-prosthesis continuously moving in space. During the BC process, the subject imagines the movement of the real limb and adapts the brain activity according to the sensory feedback. The neural adaptation in the closed-loop control results in complex and changing brain signals. Simultaneously, the decoder interprets the time-varying functional mapping between neural activity and continuous trajectory. It is crucial and challenging to accurately and adaptively track the mapping to help the subject accomplish the BC task with a stable performance. Existing Kalman Filter (KF) based decoders achieve continuous trajectory control by linearly interpreting neural firing observations into self-evolving prosthetic states. However, the linear neural-state mapping might not accurately reflect the movement intention of the subject. In this paper, we propose a novel method that allows subjects to achieve continuous brain control efficiently and stably. The proposed method incorporates a kernel reinforcement learning method into a state-observation model to decode the nonlinearly neural observation into a continuous trajectory state. The state transition function ensures the continuity of the prosthetic state. The kernel reinforcement learning allows the quick adaptation of the nonlinear neural-movement mapping during the BC process. The proposed method is tested in an online brain control reaching task for rats. Compared with KF, our method achieved more successful trials, faster response time, shorter inter-trial time, and remained stable over days. These results demonstrate that the proposed method is an efficient tool to assist subjects in brain control tasks.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

Covariant Cluster Transfer for Kernel Reinforcement Learning in Brain-Machine Interface

Brain-Machine Interface (BMI) provides a promising way to help disabled people restore their motor functions. The patients are able to control the external devices directly from their neural signals by the decoder. Due to various reasons such as mental fatigue and distraction, the distribution of the neural signals might change, which might lead to poor performance for the decoder. In this case, we need to calibrate the parameters before each session, which needs the professionals to label the data and is not convenient for the patient's usage at home. In this paper, we propose a covariant cluster transfer mechanism for the kernel reinforcement learning (RL) algorithm to speed up the adaptation across sessions. The parameters of the decoder will adaptively change according to a reward signal, which could be easily set by the patient. More importantly, we cluster the neural patterns in previous sessions. The cluster represents the conditional distribution from neural patterns to actions. When a distinct neural pattern appears in the new session, the nearest cluster will be transferred. In this way, the knowledge from the old session could be utilized to accelerate the learning in the new session. Our proposed algorithm is tested on the simulated neural data where the neural signal's distribution differs across sessions. Compared with the training from random initialization and a weight transfer policy, our proposed cluster transfer mechanism maintains a significantly higher success rate and a faster adaptation when the conditional distribution from neural signals to actions remains similar.