Decoding fMRI brain states in real-time

Guiding principles and considerations for designing a well-structured curriculum for the brain-computer interface major based on the multidisciplinary nature of brain-computer interface

Brain-computer interface (BCI) is a novel human-computer interaction technology, and its rapid development has led to a growing demand for skilled BCI professionals, culminating in the emergence of the BCI major. Despite its significance, there is limited literature addressing the curriculum design for this emerging major. This paper seeks to bridge this gap by proposing and discussing a curricular framework for the BCI major, based on the inherently multidisciplinary nature of BCI research and development. The paper begins by elucidating the primary factors behind the emergence of the BCI major, the increasing demand for both medical and non-medical applications of BCI, and the corresponding need for specialized talent. It then delves into the multidisciplinary nature of BCI research and offers principles for curriculum design to address this nature. Based on these principles, the paper provides detailed suggestions for structuring a BCI curriculum. Finally, it discusses the challenges confronting the development of the BCI major, including the lack of consensus and international collaboration in the construction of the BCI major, as well as the inadequacy or lack of teaching materials. Future work needs to improve the curriculum design of the BCI major from a competency-oriented perspective. It is expected that this paper will provide a reference for the curriculum design and construction of the BCI major.

Regulation of craving for real-time fMRI neurofeedback based on individual classification

In previous real-time functional magnetic resonance imaging neurofeedback (rtfMRI-NF) studies on smoking craving, the focus has been on within-region activity or between-region connectivity, neglecting the potential predictive utility of broader network activity. Moreover, there is debate over the use and relative predictive power of individual-specific and group-level classifiers. This study aims to further advance rtfMRI-NF for substance use disorders by using whole-brain rtfMRI-NF to assess smoking craving-related brain patterns, evaluate the performance of group-level or individual-level classification (n = 31) and evaluate the performance of an optimized classifier across repeated NF runs. Using real-time individual-level classifiers derived from whole-brain support vector machines, we found that classification accuracy between crave and no-crave conditions and between repeated NF runs increased across repeated runs at both individual and group levels. In addition, individual-level accuracy was significantly greater than group-level accuracy, highlighting the potential increased utility of an individually trained whole-brain classifier for volitional control over brain patterns to regulate smoking craving. This study provides evidence supporting the feasibility of using whole-brain rtfMRI-NF to modulate smoking craving-related brain responses and the potential for learning individual strategies through optimization across repeated feedback runs. This article is part of the theme issue 'Neurofeedback: new territories and neurocognitive mechanisms of endogenous neuromodulation'.

Closed-loop fMRI at the mesoscopic scale of columns and layers: Can we do it and why would we want to?

Technological advances in fMRI including ultra-high magnetic fields (≥ 7 T) and acquisition methods that increase spatial specificity have paved the way for studies of the human cortex at the scale of layers and columns. This mesoscopic scale promises an improved mechanistic understanding of human cortical function so far only accessible to invasive animal neurophysiology. In recent years, an increasing number of studies have applied such methods to better understand the cortical function in perception and cognition. This future perspective article asks whether closed-loop fMRI studies could equally benefit from these methods to achieve layer and columnar specificity. We outline potential applications and discuss the conceptual and concrete challenges, including data acquisition and volitional control of mesoscopic brain activity. We anticipate an important role of fMRI with mesoscopic resolution for closed-loop fMRI and neurofeedback, yielding new insights into brain function and potentially clinical applications.This article is part of the theme issue 'Neurofeedback: new territories and neurocognitive mechanisms of endogenous neuromodulation'.

Immersive Virtual Reality-Based Methods for Assessing Executive Functioning: Systematic Review

BACKGROUND

Neuropsychological assessments traditionally include tests of executive functioning (EF) because of its critical role in daily activities and link to mental disorders. Established traditional EF assessments, although robust, lack ecological validity and are limited to single cognitive processes. These methods, which are suitable for clinical populations, are less informative regarding EF in healthy individuals. With these limitations in mind, immersive virtual reality (VR)-based assessments of EF have garnered interest because of their potential to increase test sensitivity, ecological validity, and neuropsychological assessment accessibility.

OBJECTIVE

This systematic review aims to explore the literature on immersive VR assessments of EF focusing on (1) EF components being assessed, (2) how these assessments are validated, and (3) strategies for monitoring potential adverse (cybersickness) and beneficial (immersion) effects.

METHODS

EBSCOhost, Scopus, and Web of Science were searched in July 2022 using keywords that reflected the main themes of VR, neuropsychological tests, and EF. Articles had to be peer-reviewed manuscripts written in English and published after 2013 that detailed empirical, clinical, or proof-of-concept studies in which a virtual environment using a head-mounted display was used to assess EF in an adult population. A tabular synthesis method was used in which validation details from each study, including comparative assessments and scores, were systematically organized in a table. The results were summed and qualitatively analyzed to provide a comprehensive overview of the findings.

RESULTS

The search retrieved 555 unique articles, of which 19 (3.4%) met the inclusion criteria. The reviewed studies encompassed EF and associated higher-order cognitive functions such as inhibitory control, cognitive flexibility, working memory, planning, and attention. VR assessments commonly underwent validation against gold-standard traditional tasks. However, discrepancies were observed, with some studies lacking reported a priori planned correlations, omitting detailed descriptions of the EF constructs evaluated using the VR paradigms, and frequently reporting incomplete results. Notably, only 4 of the 19 (21%) studies evaluated cybersickness, and 5 of the 19 (26%) studies included user experience assessments.

CONCLUSIONS

Although it acknowledges the potential of VR paradigms for assessing EF, the evidence has limitations. The methodological and psychometric properties of the included studies were inconsistently addressed, raising concerns about their validity and reliability. Infrequent monitoring of adverse effects such as cybersickness and considerable variability in sample sizes may limit interpretation and hinder psychometric evaluation. Several recommendations are proposed to improve the theory and practice of immersive VR assessments of EF. Future studies should explore the integration of biosensors with VR systems and the capabilities of VR in the context of spatial navigation assessments. Despite considerable promise, the systematic and validated implementation of VR assessments is essential for ensuring their practical utility in real-world applications.

Two is better? combining EEG and fMRI for BCI and neurofeedback: a systematic review

Electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) are two commonly used non-invasive techniques for measuring brain activity in neuroscience and brain-computer interfaces (BCI).Objective. In this review, we focus on the use of EEG and fMRI in neurofeedback (NF) and discuss the challenges of combining the two modalities to improve understanding of brain activity and achieve more effective clinical outcomes. Advanced technologies have been developed to simultaneously record EEG and fMRI signals to provide a better understanding of the relationship between the two modalities. However, the complexity of brain processes and the heterogeneous nature of EEG and fMRI present challenges in extracting useful information from the combined data.Approach. We will survey existing EEG-fMRI combinations and recent studies that exploit EEG-fMRI in NF, highlighting the experimental and technical challenges.Main results. We made a classification of the different combination of EEG-fMRI for NF, we provide a review of multimodal analysis methods for EEG-fMRI features. We also survey the current state of research on EEG-fMRI in the different existing NF paradigms. Finally, we also identify some of the remaining challenges in this field.Significance. By exploring EEG-fMRI combinations in NF, we are advancing our knowledge of brain function and its applications in clinical settings. As such, this review serves as a valuable resource for researchers, clinicians, and engineers working in the field of neural engineering and rehabilitation, highlighting the promising future of EEG-fMRI-based NF.

Closing the loop between brain and electrical stimulation: towards precision neuromodulation treatments

One of the most critical challenges in using noninvasive brain stimulation (NIBS) techniques for the treatment of psychiatric and neurologic disorders is inter- and intra-individual variability in response to NIBS. Response variations in previous findings suggest that the one-size-fits-all approach does not seem the most appropriate option for enhancing stimulation outcomes. While there is a growing body of evidence for the feasibility and effectiveness of individualized NIBS approaches, the optimal way to achieve this is yet to be determined. Transcranial electrical stimulation (tES) is one of the NIBS techniques showing promising results in modulating treatment outcomes in several psychiatric and neurologic disorders, but it faces the same challenge for individual optimization. With new computational and methodological advances, tES can be integrated with real-time functional magnetic resonance imaging (rtfMRI) to establish closed-loop tES-fMRI for individually optimized neuromodulation. Closed-loop tES-fMRI systems aim to optimize stimulation parameters based on minimizing differences between the model of the current brain state and the desired value to maximize the expected clinical outcome. The methodological space to optimize closed-loop tES fMRI for clinical applications includes (1) stimulation vs. data acquisition timing, (2) fMRI context (task-based or resting-state), (3) inherent brain oscillations, (4) dose-response function, (5) brain target trait and state and (6) optimization algorithm. Closed-loop tES-fMRI technology has several advantages over non-individualized or open-loop systems to reshape the future of neuromodulation with objective optimization in a clinically relevant context such as drug cue reactivity for substance use disorder considering both inter and intra-individual variations. Using multi-level brain and behavior measures as input and desired outcomes to individualize stimulation parameters provides a framework for designing personalized tES protocols in precision psychiatry.

Talking with hands and feet: Selective somatosensory attention and fMRI enable robust and convenient brain-based communication

In brain-based communication, voluntarily modulated brain signals (instead of motor output) are utilized to interact with the outside world. The possibility to circumvent the motor system constitutes an important alternative option for severely paralyzed. Most communication brain-computer interface (BCI) paradigms require intact visual capabilities and impose a high cognitive load, but for some patients, these requirements are not given. In these situations, a better-suited, less cognitively demanding information-encoding approach may exploit auditorily-cued selective somatosensory attention to vibrotactile stimulation. Here, we propose, validate and optimize a novel communication-BCI paradigm using differential fMRI activation patterns evoked by selective somatosensory attention to tactile stimulation of the right hand or left foot. Using cytoarchitectonic probability maps and multi-voxel pattern analysis (MVPA), we show that the locus of selective somatosensory attention can be decoded from fMRI-signal patterns in (especially primary) somatosensory cortex with high accuracy and reliability, with the highest classification accuracy (85.93%) achieved when using Brodmann area 2 (SI-BA2) at a probability level of 0.2. Based on this outcome, we developed and validated a novel somatosensory attention-based yes/no communication procedure and demonstrated its high effectiveness even when using only a limited amount of (MVPA) training data. For the BCI user, the paradigm is straightforward, eye-independent, and requires only limited cognitive functioning. In addition, it is BCI-operator friendly given its objective and expertise-independent procedure. For these reasons, our novel communication paradigm has high potential for clinical applications.

BolT: Fused window transformers for fMRI time series analysis

Deep-learning models have enabled performance leaps in analysis of high-dimensional functional MRI (fMRI) data. Yet, many previous methods are suboptimally sensitive for contextual representations across diverse time scales. Here, we present BolT, a blood-oxygen-level-dependent transformer model, for analyzing multi-variate fMRI time series. BolT leverages a cascade of transformer encoders equipped with a novel fused window attention mechanism. Encoding is performed on temporally-overlapped windows within the time series to capture local representations. To integrate information temporally, cross-window attention is computed between base tokens in each window and fringe tokens from neighboring windows. To gradually transition from local to global representations, the extent of window overlap and thereby number of fringe tokens are progressively increased across the cascade. Finally, a novel cross-window regularization is employed to align high-level classification features across the time series. Comprehensive experiments on large-scale public datasets demonstrate the superior performance of BolT against state-of-the-art methods. Furthermore, explanatory analyses to identify landmark time points and regions that contribute most significantly to model decisions corroborate prominent neuroscientific findings in the literature.

Inducing a mental context for associative memory formation with real-time fMRI neurofeedback

Memory, one of the hallmarks of human cognition, can be modified when humans voluntarily modulate neural population activity using neurofeedback. However, it is currently unknown whether neurofeedback can influence the integration of memories, and whether memory is facilitated or impaired after such neural perturbation. In this study, participants memorized objects while we provided them with abstract neurofeedback based on their brain activity patterns in the ventral visual stream. This neurofeedback created an implicit face or house context in the brain while memorizing the objects. The results revealed that participants created associations between each memorized object and its implicit context solely due to the neurofeedback manipulation. Our findings shed light onto how memory formation can be influenced by synthetic memory tags with neurofeedback and advance our understanding of mnemonic processing.

A real-time fMRI neurofeedback system for the clinical alleviation of depression with a subject-independent classification of brain states: A proof of principle study

Most clinical neurofeedback studies based on functional magnetic resonance imaging use the patient's own neural activity as feedback. The objective of this study was to create a subject-independent brain state classifier as part of a real-time fMRI neurofeedback (rt-fMRI NF) system that can guide patients with depression in achieving a healthy brain state, and then to examine subsequent clinical changes. In a first step, a brain classifier based on a support vector machine (SVM) was trained from the neural information of happy autobiographical imagery and motor imagery blocks received from a healthy female participant during an MRI session. In the second step, 7 right-handed female patients with mild or moderate depressive symptoms were trained to match their own neural activity with the neural activity corresponding to the “happiness emotional brain state” of the healthy participant. The training (4 training sessions over 2 weeks) was carried out using the rt-fMRI NF system guided by the brain-state classifier we had created. Thus, the informative voxels previously obtained in the first step, using SVM classification and Effect Mapping, were used to classify the Blood-Oxygen-Level Dependent (BOLD) activity of the patients and converted into real-time visual feedback during the neurofeedback training runs. Improvements in the classifier accuracy toward the end of the training were observed in all the patients [Session 4–1 Median = 6.563%; Range = 4.10–27.34; Wilcoxon Test (0), 2-tailed p = 0.031]. Clinical improvement also was observed in a blind standardized clinical evaluation [HDRS CE2-1 Median = 7; Range 2 to 15; Wilcoxon Test (0), 2-tailed p = 0.016], and in self-report assessments [BDI-II CE2-1 Median = 8; Range 1–15; Wilcoxon Test (0), 2-tailed p = 0.031]. In addition, the clinical improvement was still present 10 days after the intervention [BDI-II CE3-2_Median = 0; Range −1 to 2; Wilcoxon Test (0), 2-tailed p = 0.50/ HDRS CE3-2 Median = 0; Range −1 to 2; Wilcoxon Test (0), 2-tailed p = 0.625]. Although the number of participants needs to be increased and a control group included to confirm these findings, the results suggest a novel option for neural modulation and clinical alleviation in depression using noninvasive stimulation technologies.

Neural Mechanisms of Facial Emotion Recognition in Autism: Distinct Roles for Anterior Cingulate and dlPFC

OBJECTIVE

The present study sought to measure and internally validate neural markers of facial emotion recognition (FER) in adolescents and young adults with ASD to inform targeted intervention.

METHOD

We utilized fMRI to measure patterns of brain activity among individuals with ASD (N = 21) and matched controls (CON; N = 20) 2 s prior to judgments about the identity of six distinct facial emotions (happy, sad, angry, surprised, fearful, disgust).

RESULTS

Predictive modeling of fMRI data (support vector classification; SVC) identified mechanistic roles for brain regions that forecasted correct and incorrect identification of facial emotion as well as sources of errors over these decisions. BOLD signal activation in bilateral insula, anterior cingulate (ACC) and right dorsolateral prefrontal cortex (dlPFC) preceded accurate FER in both controls and ASD. Predictive modeling utilizing SVC confirmed the utility of ACC in forecasting correct decisions in controls but not ASD, and further indicated that a region within the right dlPFC was the source of a type 1 error signal in ASD (i.e. neural marker reflecting an impending correct judgment followed by an incorrect behavioral response) approximately two seconds prior to emotion judgments during fMRI.

CONCLUSIONS

ACC forecasted correct decisions only among control participants. Right dlPFC was the source of a false-positive signal immediately prior to an error about the nature of a facial emotion in adolescents and young adults with ASD, potentially consistent with prior work indicating that dlPFC may play a role in attention to and regulation of emotional experience.

Using causal methods to map symptoms to brain circuits in neurodevelopment disorders: moving from identifying correlates to developing treatments

A wide variety of model systems and experimental techniques can provide insight into the structure and function of the human brain in typical development and in neurodevelopmental disorders. Unfortunately, this work, whether based on manipulation of animal models or observational and correlational methods in humans, has a high attrition rate in translating scientific discovery into practicable treatments and therapies for neurodevelopmental disorders.With new computational and neuromodulatory approaches to interrogating brain networks, opportunities exist for "bedside-to bedside-translation" with a potentially shorter path to therapeutic options. Specifically, methods like lesion network mapping can identify brain networks involved in the generation of complex symptomatology, both from acute onset lesion-related symptoms and from focal developmental anomalies. Traditional neuroimaging can examine the generalizability of these findings to idiopathic populations, while non-invasive neuromodulation techniques such as transcranial magnetic stimulation provide the ability to do targeted activation or inhibition of these specific brain regions and networks. In parallel, real-time functional MRI neurofeedback also allow for endogenous neuromodulation of specific targets that may be out of reach for transcranial exogenous methods.Discovery of novel neuroanatomical circuits for transdiagnostic symptoms and neuroimaging-based endophenotypes may now be feasible for neurodevelopmental disorders using data from cohorts with focal brain anomalies. These novel circuits, after validation in large-scale highly characterized research cohorts and tested prospectively using noninvasive neuromodulation and neurofeedback techniques, may represent a new pathway for symptom-based targeted therapy.

fMRI Brain Decoding and Its Applications in Brain-Computer Interface: A Survey

Brain neural activity decoding is an important branch of neuroscience research and a key technology for the brain-computer interface (BCI). Researchers initially developed simple linear models and machine learning algorithms to classify and recognize brain activities. With the great success of deep learning on image recognition and generation, deep neural networks (DNN) have been engaged in reconstructing visual stimuli from human brain activity via functional magnetic resonance imaging (fMRI). In this paper, we reviewed the brain activity decoding models based on machine learning and deep learning algorithms. Specifically, we focused on current brain activity decoding models with high attention: variational auto-encoder (VAE), generative confrontation network (GAN), and the graph convolutional network (GCN). Furthermore, brain neural-activity-decoding-enabled fMRI-based BCI applications in mental and psychological disease treatment are presented to illustrate the positive correlation between brain decoding and BCI. Finally, existing challenges and future research directions are addressed.

Is it time to put rest to rest?

The so-called resting state, in which participants lie quietly with no particular inputs or outputs, represented a paradigm shift from conventional task-based studies in human neuroimaging. Our foray into rest was fruitful from both a scientific and methodological perspective, but at this point, how much more can we learn from rest on its own? While rest still dominates in many subfields, data from tasks have empirically demonstrated benefits, as well as the potential to provide insights about the mind in addition to the brain. I argue that we can accelerate progress in human neuroscience by de-emphasizing rest in favor of more grounded experiments, including promising integrated designs that respect the prominence of self-generated activity while offering enhanced control and interpretability.

Decoding Three Different Preference Levels of Consumers Using Convolutional Neural Network: A Functional Near-Infrared Spectroscopy Study

This study decodes consumers' preference levels using a convolutional neural network (CNN) in neuromarketing. The classification accuracy in neuromarketing is a critical factor in evaluating the intentions of the consumers. Functional near-infrared spectroscopy (fNIRS) is utilized as a neuroimaging modality to measure the cerebral hemodynamic responses. In this study, a specific decoding structure, called CNN-based fNIRS-data analysis, was designed to achieve a high classification accuracy. Compared to other methods, the automated characteristics, constant training of the dataset, and learning efficiency of the proposed method are the main advantages. The experimental procedure required eight healthy participants (four female and four male) to view commercial advertisement videos of different durations (15, 30, and 60 s). The cerebral hemodynamic responses of the participants were measured. To compare the preference classification performances, CNN was utilized to extract the most common features, including the mean, peak, variance, kurtosis, and skewness. Considering three video durations, the average classification accuracies of 15, 30, and 60 s videos were 84.3, 87.9, and 86.4%, respectively. Among them, the classification accuracy of 87.9% for 30 s videos was the highest. The average classification accuracies of three preferences in females and males were 86.2 and 86.3%, respectively, showing no difference in each group. By comparing the classification performances in three different combinations (like vs. so-so, like vs. dislike, and so-so vs. dislike) between two groups, male participants were observed to have targeted preferences for commercial advertising, and the classification performance 88.4% between "like" vs. "dislike" out of three categories was the highest. Finally, pairwise classification performance are shown as follows: For female, 86.1% (like vs. so-so), 87.4% (like vs. dislike), 85.2% (so-so vs. dislike), and for male 85.7, 88.4, 85.1%, respectively.

A systematic review of fMRI neurofeedback reporting and effects in clinical populations

Real-time fMRI-based neurofeedback is a relatively young field with a potential to impact the currently available treatments of various disorders. In order to evaluate the evidence of clinical benefits and investigate how consistently studies report their methods and results, an exhaustive search of fMRI neurofeedback studies in clinical populations was performed. Reporting was evaluated using a limited number of Consensus on the reporting and experimental design of clinical and cognitive-behavioral neurofeedback studies (CRED-NF checklist) items, which was, together with a statistical power and sensitivity calculation, used to also evaluate the existing evidence of the neurofeedback benefits on clinical measures. The 62 found studies investigated regulation abilities and/or clinical benefits in a wide range of disorders, but with small sample sizes and were therefore unable to detect small effects. Most points from the CRED-NF checklist were adequately reported by the majority of the studies, but some improvements are suggested for the reporting of group comparisons and relations between regulation success and clinical benefits. To establish fMRI neurofeedback as a clinical tool, more emphasis should be placed in the future on using larger sample sizes determined through a priori power calculations and standardization of procedures and reporting.

fMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations

Deep-learning methods based on deep neural networks (DNNs) have recently been successfully utilized in the analysis of neuroimaging data. A convolutional neural network (CNN) is a type of DNN that employs a convolution kernel that covers a local area of the input sample and moves across the sample to provide a feature map for the subsequent layers. In our study, we hypothesized that a 3D-CNN model with down-sampling operations such as pooling and/or stride would have the ability to extract robust feature maps from the shifted and scaled neuronal activations in a single functional MRI (fMRI) volume for the classification of task information associated with that volume. Thus, the 3D-CNN model would be able to ameliorate the potential misalignment of neuronal activations and over-/under-activation in local brain regions caused by imperfections in spatial alignment algorithms, confounded by variability in blood-oxygenation-level-dependent (BOLD) responses across sessions and/or subjects. To this end, the fMRI volumes acquired from four sensorimotor tasks (left-hand clenching, right-hand clenching, auditory attention, and visual stimulation) were used as input for our 3D-CNN model to classify task information using a single fMRI volume. The classification performance of the 3D-CNN was systematically evaluated using fMRI volumes obtained from various minimal preprocessing scenarios applied to raw fMRI volumes that excluded spatial normalization to a template and those obtained from full preprocessing that included spatial normalization. Alternative classifier models such as the 1D fully connected DNN (1D-fcDNN) and support vector machine (SVM) were also used for comparison. The classification performance was also assessed for several k-fold cross-validation (CV) schemes, including leave-one-subject-out CV (LOOCV). Overall, the classification results of the 3D-CNN model were superior to that of the 1D-fcDNN and SVM models. When using the fully-processed fMRI volumes with LOOCV, the mean error rates (± the standard error of the mean) for the 3D-CNN, 1D-fcDNN, and SVM models were 2.1% (± 0.9), 3.1% (± 1.2), and 4.1% (± 1.5), respectively (p = 0.041 from a one-way ANOVA). The error rates for 3-fold CV were higher (2.4% ± 1.0, 4.2% ± 1.3, and 10.1% ± 2.0; p < 0.0003 from a one-way ANOVA). The mean error rates also increased considerably using the raw fMRI 3D volume data without preprocessing (26.2% for the 3D-CNN, 75.0% for the 1D-fcDNN, and 75.0% for the SVM). Furthermore, the ability of the pre-trained 3D-CNN model to handle shifted and scaled neuronal activations was demonstrated in an online scenario for five-class classification (i.e., four sensorimotor tasks and the resting state) using the real-time fMRI of three participants. The resulting classification accuracy was 78.5% (± 1.4), 26.7% (± 5.9), and 21.5% (± 3.1) for the 3D-CNN, 1D-fcDNN, and SVM models, respectively. The superior performance of the 3D-CNN compared to the 1D-fcDNN was verified by analyzing the resulting feature maps and convolution filters that handled the shifted and scaled neuronal activations and by utilizing an independent public dataset from the Human Connectome Project.

Quality and denoising in real-time functional magnetic resonance imaging neurofeedback: A methods review

Neurofeedback training using real-time functional magnetic resonance imaging (rtfMRI-NF) allows subjects voluntary control of localised and distributed brain activity. It has sparked increased interest as a promising non-invasive treatment option in neuropsychiatric and neurocognitive disorders, although its efficacy and clinical significance are yet to be determined. In this work, we present the first extensive review of acquisition, processing and quality control methods available to improve the quality of the neurofeedback signal. Furthermore, we investigate the state of denoising and quality control practices in 128 recently published rtfMRI-NF studies. We found: (a) that less than a third of the studies reported implementing standard real-time fMRI denoising steps, (b) significant room for improvement with regards to methods reporting and (c) the need for methodological studies quantifying and comparing the contribution of denoising steps to the neurofeedback signal quality. Advances in rtfMRI-NF research depend on reproducibility of methods and results. Notably, a systematic effort is needed to build up evidence that disentangles the various mechanisms influencing neurofeedback effects. To this end, we recommend that future rtfMRI-NF studies: (a) report implementation of a set of standard real-time fMRI denoising steps according to a proposed COBIDAS-style checklist (https://osf.io/kjwhf/), (b) ensure the quality of the neurofeedback signal by calculating and reporting community-informed quality metrics and applying offline control checks and (c) strive to adopt transparent principles in the form of methods and data sharing and support of open-source rtfMRI-NF software. Code and data for reproducibility, as well as an interactive environment to explore the study data, can be accessed at https://github.com/jsheunis/quality-and-denoising-in-rtfmri-nf.

Neuromodulation of brain activation associated with addiction: A review of real-time fMRI neurofeedback studies

Real-time functional magnetic resonance imaging neurofeedback (rtfMRI-nf) has emerged in recent years as an imaging modality used to examine volitional control over targeted brain activity. rtfMRI-nf has also been applied clinically as a way to train individuals to self-regulate areas of the brain, or circuitry, involved in various disorders. One such application of rtfMRI-nf has been in the domain of addictive behaviors, including substance use. Given the pervasiveness of substance use and the challenges of existing treatments to sustain abstinence, rtfMRI-nf has been identified as a promising treatment tool. rtfMRI-nf has also been used in basic science research in order to test the ability to modulate brain function involved in addiction. This review focuses first on providing an overview of recent rtfMRI-nf studies in substance-using populations, specifically nicotine, alcohol, and cocaine users, aimed at reducing craving-related brain activation. Next, rtfMRI-nf studies targeting reward responsivity and emotion regulation in healthy samples are reviewed in order to examine the extent to which areas of the brain involved in addiction can be self-regulated using neurofeedback. We propose that future rtfMRI-nf studies could be strengthened by improvements to study design, sample selection, and more robust strategies in the development and assessment of rtfMRI-nf as a clinical treatment. Recommendations for ways to accomplish these improvements are provided. rtfMRI-nf holds much promise as an imaging modality that can directly target key brain regions involved in addiction, however additional studies are needed in order to establish rtfMRI-nf as an effective, and practical, treatment for addiction.

Improving Real-Time Brain State Classification of Motor Imagery Tasks During Neurofeedback Training

In this study, we investigated the effect of the dynamic changes in brain activation during neurofeedback training in the classification of the different brain states associated with the target tasks. We hypothesized that ongoing activation patterns could change during neurofeedback session due to learning effects and, in the process, could affect the performance of brain state classifiers trained using data obtained prior to the session. Using a motor imagery paradigm, we then examined the application of an incremental training approach where classifiers were continuously updated in order to account for these activation changes. Our results confirmed our hypothesis that neurofeedback training could be associated with dynamic changes in brain activation characterized by an initially more widespread brain activation followed by a more focused and localized activation pattern. By continuously updating the trained classifiers after each feedback run, significant improvement in accurately classifying the different brain states associated with the target motor imagery tasks was achieved. These findings suggest the importance of taking into account brain activation changes during neurofeedback in order to provide more reliable and accurate feedback information to the participants, which is critical for an effective neurofeedback application.

Support Vector Machines and Affective Science

Support vector machines (SVMs) are being used increasingly in affective science as a data-driven classification method and feature reduction technique. Whereas traditional statistical methods typically compare group averages on selected variables, SVMs use a predictive algorithm to learn multivariate patterns that optimally discriminate between groups. In this review, we provide a framework for understanding the methods of SVM-based analyses and summarize the findings of seminal studies that use SVMs for classification or data reduction in the behavioral and neural study of emotion and affective disorders. We conclude by discussing promising directions and potential applications of SVMs in future research in affective science.

A computational paradigm for real-time MEG neurofeedback for dynamic allocation of spatial attention

Background

Neurofeedback aids volitional control of one’s own brain activity using non-invasive recordings of brain activity. The applications of neurofeedback include improvement of cognitive performance and treatment of various psychiatric and neurological disorders. During real-time magnetoencephalography (rt-MEG), sensor-level or source-localized brain activity is measured and transformed into a visual feedback cue to the subject. Recent real-time fMRI (rt-fMRI) neurofeedback studies have used pattern recognition techniques to decode and train a brain state to link brain activities and cognitive behaviors. Here, we utilize the real-time decoding technique similar to ones employed in rt-fMRI to analyze time-varying rt-MEG signals.

Results

We developed a novel rt-MEG method, state-based neurofeedback (sb-NFB), to decode a time-varying brain state, a state signal, from which timings are extracted for neurofeedback training. The approach is entirely data-driven: it uses sensor-level oscillatory activity to find relevant features that best separate the targeted brain states. In a psychophysical task of spatial attention switching, we trained five young, healthy subjects using the sb-NFB method to decrease the time necessary for switch spatial attention from one visual hemifield to the other (referred to as switch time). Training resulted in a decrease in switch time with training. We saw that the activity targeted by the training involved proportional changes in alpha and beta-band oscillations, in sensors at the occipital and parietal regions. We also found that the state signal that encodes whether subjects attend to the left or right visual field effectively switches consistently with the task.

Conclusion

We demonstrated the use of the sb-NFB method when the subject learns to increase the speed of shifting covert spatial attention from one visual field to the other. The sb-NFB method can target timing features that would otherwise also include extraneous features such as visual detection and motor response in a simple reaction time task.

Fast detection and reduction of local transient artifacts in resting-state fMRI

Image quality control (QC) is a critical and computationally intensive component of functional magnetic resonance imaging (fMRI). Artifacts caused by physiologic signals or hardware malfunctions are usually identified and removed during data processing offline, well after scanning sessions are complete. A system with the computational efficiency to identify and remove artifacts during image acquisition would permit rapid adjustment of protocols as issues arise during experiments. To improve the speed and accuracy of QC and functional image correction, we developed Fast Anatomy-Based Image Correction (Fast ANATICOR) with newly implemented nuisance models and an improved pipeline. We validated its performance on a dataset consisting of normal scans and scans containing known hardware-driven artifacts. Fast ANATICOR's increased processing speed may make real-time QC and image correction feasible as compared with the existing offline method.

A Guide to Literature Informed Decisions in the Design of Real Time fMRI Neurofeedback Studies: A Systematic Review

Background: Although biofeedback using electrophysiology has been explored extensively, the approach of using neurofeedback corresponding to hemodynamic response is a relatively young field. Real time functional magnetic resonance imaging-based neurofeedback (rt-fMRI-NF) uses sensory feedback to operantly reinforce patterns of neural response. It can be used, for example, to alter visual perception, increase brain connectivity, and reduce depression symptoms. Within recent years, interest in rt-fMRI-NF in both research and clinical contexts has expanded considerably. As such, building a consensus regarding best practices is of great value. Objective: This systematic review is designed to describe and evaluate the variations in methodology used in previous rt-fMRI-NF studies to provide recommendations for rt-fMRI-NF study designs that are mostly likely to elicit reproducible and consistent effects of neurofeedback. Methods: We conducted a database search for fMRI neurofeedback papers published prior to September 26th, 2019. Of 558 studies identified, 146 met criteria for inclusion. The following information was collected from each study: sample size and type, task used, neurofeedback calculation, regulation procedure, feedback, whether feedback was explicitly related to changing brain activity, feedback timing, control group for active neurofeedback, how many runs and sessions of neurofeedback, if a follow-up was conducted, and the results of neurofeedback training. Results: rt-fMRI-NF is typically upregulation practice based on hemodynamic response from a specific region of the brain presented using a continually updating thermometer display. Most rt-fMRI-NF studies are conducted in healthy samples and half evaluate its effect on immediate changes in behavior or affect. The most popular control group method is to provide sham signal from another region; however, many studies do not compare use a comparison group. Conclusions: We make several suggestions for designs of future rt-fMRI-NF studies. Researchers should use feedback calculation methods that consider neural response across regions (i.e., SVM or connectivity), which should be conveyed as intermittent, auditory feedback. Participants should be given explicit instructions and should be assessed on individual differences. Future rt-fMRI-NF studies should use clinical samples; effectiveness of rt-fMRI-NF should be evaluated on clinical/behavioral outcomes at follow-up time points in comparison to both a sham and no feedback control group.

Real-time fMRI for brain-computer interfacing

Brain-computer interfaces (BCIs) based on functional magnetic resonance imaging (fMRI) provide an important complement to other noninvasive BCIs. While fMRI has several disadvantages (being nonportable, methodologically challenging, costly, and noisy), it is the only method providing high spatial resolution whole-brain coverage of brain activation. These properties allow relating mental activities to specific brain regions and networks providing a transparent scheme for BCI users to encode information and for real-time fMRI BCI systems to decode the intents of the user. Various mental activities have been used successfully in fMRI BCIs so far that can be classified into the four categories: (a) higher-order cognitive tasks (e.g., mental calculation), (b) covert language-related tasks (e.g., mental speech and mental singing), (c) imagery tasks (motor, visual, auditory, tactile, and emotion imagery), and (d) selective attention tasks (visual, auditory, and tactile attention). While the ultimate spatial and temporal resolution of fMRI BCIs is limited by the physiologic properties of the hemodynamic response, technical and analytical advances will likely lead to substantially improved fMRI BCIs in the future using, for example, decoding of imagined letter shapes at 7T as the basis for more "natural" communication BCIs.

Precision diagnostics based on machine learning-derived imaging signatures

The complexity of modern multi-parametric MRI has increasingly challenged conventional interpretations of such images. Machine learning has emerged as a powerful approach to integrating diverse and complex imaging data into signatures of diagnostic and predictive value. It has also allowed us to progress from group comparisons to imaging biomarkers that offer value on an individual basis. We review several directions of research around this topic, emphasizing the use of machine learning in personalized predictions of clinical outcome, in breaking down broad umbrella diagnostic categories into more detailed and precise subtypes, and in non-invasively estimating cancer molecular characteristics. These methods and studies contribute to the field of precision medicine, by introducing more specific diagnostic and predictive biomarkers of clinical outcome, therefore pointing to better matching of treatments to patients.

Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networks

Decoding brain functional states underlying cognitive processes from functional MRI (fMRI) data using multivariate pattern analysis (MVPA) techniques has achieved promising performance for characterizing brain activation patterns and providing neurofeedback signals. However, it remains challenging to decode subtly distinct brain states for individual fMRI data points due to varying temporal durations and dependency among different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in intrinsic functional network modeling and sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, subject-specific intrinsic functional networks (FNs) are computed from resting-state fMRI data and are used to characterize functional signals of task fMRI data with a compact representation for building brain decoding models, and LSTM RNNs are adopted to learn brain decoding mappings between functional profiles and brain states. Validation results on fMRI data from the HCP dataset have demonstrated that brain decoding models built on training data using the proposed method could learn discriminative latent feature representations and effectively distinguish subtly distinct working memory tasks of different subjects with significantly higher accuracy than conventional decoding models. Informative FNs of the brain decoding models identified as brain activation patterns of working memory tasks were largely consistent with the literature. The method also obtained promising decoding performance on motor and social cognition tasks. Our results suggest that LSTM RNNs in conjunction with FNs could build interpretable, highly accurate brain decoding models.

Amygdala Modulation During Emotion Regulation Training With fMRI-Based Neurofeedback

Available evidence suggests that individuals can enhance their ability to modulate brain activity in target regions, within the Emotion Regulation network, using fMRI-based neurofeedback. However, there is no systematic review that investigates the effectiveness of this method on amygdala modulation, a core region within this network. The major goal of this study was to systematically review and analyze the effects of real-time fMRI-Neurofeedback concerning the neuromodulation of the amygdala during Emotion Regulation training. A search was performed in PubMed, Science Direct, and Web of Science with the following key terms: ≪(“neurofeedback” or “neuro feedback” or “neuro-feedback”) and (“emotion regulation”) and (fMRI OR “functional magnetic resonance”),≫ and afterwards two additional searches were performed, replacing the term “emotion regulation” for “amygdala” and “neurofeedback” for “feedback.” Of the 531 identified articles, only 19 articles reported results of amygdala modulation during Emotional Regulation training through rtfMRI-NF, using healthy participants or patients, in original research articles. The results, systematically reviewed here, provide evidence for amygdala's modulation during rtfMRI-NF training, although studies' heterogeneity precluded a quantitative meta-analysis—the included studies relied on different outcome measures to infer the success of neurofeedback intervention. Thus, a qualitative analysis was done instead. We identified critical features influencing inference on the quality of the intervention as: the inclusion of a Practice Run, a Transfer Run and a Control Group in the protocol, and to choose adequate Emotion Regulation strategies—in particular, the effective recall of autobiographic memories. Surprisingly, the Regulated vs. Control Condition was lacking in most of the studies, precluding valid inference of amygdala neuromodulation within Session. The best controlled studies nevertheless showed positive effects. The type of stimulus/interface did not seem critical for amygdala modulation. We also identified potential effects of lateralization of amygdala responses following Up- or Down-Regulation, and the impact of fMRI parameters for data acquisition and analysis. Despite qualitative evidence for amygdala modulation during rtfMRI-NF, there are still important limitations in the design of a clear conceptual framework of NF-training research. Future studies should focus on more homogeneous guidelines concerning design, protocol structure and, particularly, harmonized outcome measures to provide quantitative estimates of neuromodulatory effects in the amygdala.

Targeting dynamic facial processing mechanisms in superior temporal sulcus using a novel fMRI neurofeedback target

The superior temporal sulcus (STS) encompasses a complex set of regions involved in a wide range of cognitive functions. To understand its functional properties, neuromodulation approaches such brain stimulation or neurofeedback can be used. We investigated whether the posterior STS (pSTS), a core region in the face perception and imagery network, could be specifically identified based on the presence of dynamic facial expressions (and not just on simple motion or static face signals), and probed with neurofeedback. Recognition of facial expressions is critically impaired in autism spectrum disorder, making this region a relevant target for future clinical neurofeedback studies. We used a stringent localizer approach based on the contrast of dynamic facial expressions against static neutral faces plus moving dots. The target region had to be specifically responsive to dynamic facial expressions instead of mere motion and/or the presence of a static face. The localizer was successful in selecting this region across subjects. Neurofeedback was then performed, using this region as a target, with two novel feedback rules (mean or derivative-based, using visual or auditory interfaces). Our results provide evidence that a facial expression-selective cluster in pSTS can be identified and may represent a suitable target for neurofeedback approaches, aiming at social and emotional cognition. These findings highlight the presence of a highly selective region in STS encoding dynamic aspects of facial expressions. Future studies should elucidate its role as a mechanistic target for neurofeedback strategies in clinical disorders of social cognition such as autism.

Neu3CA-RT: A framework for real-time fMRI analysis

Real-time functional magnetic resonance imaging (rtfMRI) allows visualisation of ongoing brain activity of the subject in the scanner. Denoising algorithms aim to rid acquired data of confounding effects, enhancing the blood oxygenation level-dependent (BOLD) signal. Further image processing and analysis methods, like general linear models (GLM) or multivariate analysis, then present application-specific information to the researcher. These processes are typically applied to regions of interest but, increasingly, rtfMRI techniques extract and classify whole brain functional networks and dynamics as correlates for brain states or behaviour, particularly in neuropsychiatric and neurocognitive disorders. We present Neu³CA-RT: a Matlab-based rtfMRI analysis framework aiming to advance scientific knowledge on real-time cognitive brain activity and to promote its translation into clinical practice. Design considerations are listed based on reviewing existing rtfMRI approaches. The toolbox integrates established SPM preprocessing routines, real-time GLM mapping of fMRI data to a basis set of spatial brain networks, correlation of activity with 50 behavioural profiles from the BrainMap database, and an intuitive user interface. The toolbox is demonstrated in a task-based experiment where a subject executes visual, auditory and motor tasks inside a scanner. In three out of four experiments, resulting behavioural profiles agreed with the expected brain state.

Deep neural network predicts emotional responses of the human brain from functional magnetic resonance imaging

An artificial neural network with multiple hidden layers (known as a deep neural network, or DNN) was employed as a predictive model (DNN_p) for the first time to predict emotional responses using whole-brain functional magnetic resonance imaging (fMRI) data from individual subjects. During fMRI data acquisition, 10 healthy participants listened to 80 International Affective Digital Sound stimuli and rated their own emotions generated by each sound stimulus in terms of the arousal, dominance, and valence dimensions. The whole-brain spatial patterns from a general linear model (i.e., beta-valued maps) for each sound stimulus and the emotional response ratings were used as the input and output for the DNN_P, respectively. Based on a nested five-fold cross-validation scheme, the paired input and output data were divided into training (three-fold), validation (one-fold), and test (one-fold) data. The DNN_P was trained and optimized using the training and validation data and was tested using the test data. The Pearson's correlation coefficients between the rated and predicted emotional responses from our DNN_P model with weight sparsity optimization (mean ± standard error 0.52 ± 0.02 for arousal, 0.51 ± 0.03 for dominance, and 0.51 ± 0.03 for valence, with an input denoising level of 0.3 and a mini-batch size of 1) were significantly greater than those of DNN models with conventional regularization schemes including elastic net regularization (0.15 ± 0.05, 0.15 ± 0.06, and 0.21 ± 0.04 for arousal, dominance, and valence, respectively), those of shallow models including logistic regression (0.11 ± 0.04, 0.10 ± 0.05, and 0.17 ± 0.04 for arousal, dominance, and valence, respectively; average of logistic regression and sparse logistic regression), and those of support vector machine-based predictive models (SVM_ps; 0.12 ± 0.06, 0.06 ± 0.06, and 0.10 ± 0.06 for arousal, dominance, and valence, respectively; average of linear and non-linear SVM_ps). This difference was confirmed to be significant with a Bonferroni-corrected p-value of less than 0.001 from a one-way analysis of variance (ANOVA) and subsequent paired t-test. The weights of the trained DNN_Ps were interpreted and input patterns that maximized or minimized the output of the DNN_Ps (i.e., the emotional responses) were estimated. Based on a binary classification of each emotion category (e.g., high arousal vs. low arousal), the error rates for the DNN_P (31.2% ± 1.3% for arousal, 29.0% ± 1.7% for dominance, and 28.6% ± 3.0% for valence) were significantly lower than those for the linear SVM_P (44.7% ± 2.0%, 50.7% ± 1.7%, and 47.4% ± 1.9% for arousal, dominance, and valence, respectively) and the non-linear SVM_P (48.8% ± 2.3%, 52.2% ± 1.9%, and 46.4% ± 1.3% for arousal, dominance, and valence, respectively), as confirmed by the Bonferroni-corrected p < 0.001 from the one-way ANOVA. Our study demonstrates that the DNN_p model is able to reveal neuronal circuitry associated with human emotional processing - including structures in the limbic and paralimbic areas, which include the amygdala, prefrontal areas, anterior cingulate cortex, insula, and caudate. Our DNN_p model was also able to use activation patterns in these structures to predict and classify emotional responses to stimuli.

Common Functional Brain States Encode both Perceived Emotion and the Psychophysiological Response to Affective Stimuli

Multivariate pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) data has critically advanced the neuroanatomical understanding of affect processing in the human brain. Central to these advancements is the brain state, a temporally-succinct fMRI-derived pattern of neural activation, which serves as a processing unit. Establishing the brain state's central role in affect processing, however, requires that it predicts multiple independent measures of affect. We employed MVPA-based regression to predict the valence and arousal properties of visual stimuli sampled from the International Affective Picture System (IAPS) along with the corollary skin conductance response (SCR) for demographically diverse healthy human participants (n = 19). We found that brain states significantly predicted the normative valence and arousal scores of the stimuli as well as the attendant individual SCRs. In contrast, SCRs significantly predicted arousal only. The prediction effect size of the brain state was more than three times greater than that of SCR. Moreover, neuroanatomical analysis of the regression parameters found remarkable agreement with regions long-established by fMRI univariate analyses in the emotion processing literature. Finally, geometric analysis of these parameters also found that the neuroanatomical encodings of valence and arousal are orthogonal as originally posited by the circumplex model of dimensional emotion.

Targeted partial reconstruction for real-time fMRI with arbitrary trajectories

PURPOSE

A partial image reconstruction formalism is introduced for the targeted extraction of real-time feedback from arbitrary trajectories when full image reconstruction in real time is computationally too demanding.

METHODS

Explicit calculation and storage of linear combinations of lines of the reconstruction matrix by an incomplete basis change in spatial coordinates lead to translation of the expensive full reconstruction from a frame-wise application to a region of interest (ROI)-wise application. This step is independent from signal data and can be executed before the experiment. Subsequently, the results of the sum over fully reconstructed voxels can be evaluated directly. Data from a high-speed fMRI acquisition was used to investigate the targeted partial reconstruction of a functional ROI atlas, incorporating an intravolume dephasing correction. The same data and ROIs were used for a comparison of the time series obtained with those obtained from already existing methods for compartment-wise reconstruction. To examine real-time feasibility, the reconstruction was implemented and tested for online reconstruction performance.

RESULTS

The reconstruction yields results that are virtually identical to the standard reconstruction (i.e., the magnitude sums over the ROIs), with negligible discrepancies even after termination of the conjugate gradient algorithm at a feasible number of iterations. Notably, more discrepancies arise with existing compartment-wise reconstructions. The online real-time implementation evaluated 1 ROI within 2.8 ms in the case of a highly parallel 3D whole brain acquisition.

CONCLUSION

The high reconstruction fidelity and speed are satisfying for the exemplary application of real-time functional feedback using a highly parallel 3D whole brain acquisition.

Improvements to visual working memory performance with practice and feedback

Visual working memory capacity is estimated to be around 3-4 items, but on some trials participants fail to correctly report even a single item from the memory array. Such failures of working memory performance are surprisingly common, and participants have poor self-awareness of them. Previous work has shown that behavioral feedback can reduce the frequency of working memory failures, but the benefits of feedback disappeared immediately after it was taken away. Here, we tested whether extended practice with or without trial-by-trial feedback would lead to persistent improvements in working memory performance. Participants were assigned to one of four groups: (1) Working memory practice with feedback (2) Working memory practice without feedback (3) Crossword puzzle active control (4) No-contact control. Consistent with previous work, simple practice with a visual working memory task robustly improved working memory performance across practice sessions. However, we found only partial support for the efficacy of feedback in improving working memory performance. Practicing with feedback improved working memory performance relative to a no-feedback group for some practice sessions. However, the feedback benefits did not persist across all training sessions and did not transfer to a final test session without the feedback. Thus, the benefits of performance feedback did not persist over time. Further, we found only stimulus-specific transfer of visual working memory practice benefits. We also found that participants' metaknowledge improved with practice, but that receiving feedback about task accuracy actually slightly harmed the accuracy of concurrent metaknowledge ratings. Finally, we discuss important design considerations for future work in this area (e.g. power, expectations, and "spacing effects"). For example, we found that achieved statistical power to detect a between-groups effect declined with practice. This finding has potentially critical implications for any study using a 1-session study to calculate power for a planned multi-session study.

Higher-order Brain Areas Associated with Real-time Functional MRI Neurofeedback Training of the Somato-motor Cortex

Neurofeedback (NFB) allows subjects to learn self-regulation of neuronal brain activation based on information about the ongoing activation. The implementation of real-time functional magnetic resonance imaging (rt-fMRI) for NFB training now facilitates the investigation into underlying processes. Our study involved 16 control and 16 training right-handed subjects, the latter performing an extensive rt-fMRI NFB training using motor imagery. A previous analysis focused on the targeted primary somato-motor cortex (SMC). The present study extends the analysis to the supplementary motor area (SMA), the next higher brain area within the hierarchy of the motor system. We also examined transfer-related functional connectivity using a whole-volume psycho-physiological interaction (PPI) analysis to reveal brain areas associated with learning. The ROI analysis of the pre- and post-training fMRI data for motor imagery without NFB (transfer) resulted in a significant training-specific increase in the SMA. It could also be shown that the contralateral SMA exhibited a larger increase than the ipsilateral SMA in the training and the transfer runs, and that the right-hand training elicited a larger increase in the transfer runs than the left-hand training. The PPI analysis revealed a training-specific increase in transfer-related functional connectivity between the left SMA and frontal areas as well as the anterior midcingulate cortex (aMCC) for right- and left-hand trainings. Moreover, the transfer success was related with training-specific increase in functional connectivity between the left SMA and the target area SMC. Our study demonstrates that NFB training increases functional connectivity with non-targeted brain areas. These are associated with the training strategy (i.e., SMA) as well as with learning the NFB skill (i.e., aMCC and frontal areas). This detailed description of both the system to be trained and the areas involved in learning can provide valuable information for further optimization of NFB trainings.

Improved Volitional Recall of Motor-Imagery-Related Brain Activation Patterns Using Real-Time Functional MRI-Based Neurofeedback

Motor imagery (MI), a covert cognitive process where an action is mentally simulated but not actually performed, could be used as an effective neurorehabilitation tool for motor function improvement or recovery. Recent approaches employing brain–computer/brain–machine interfaces to provide online feedback of the MI during rehabilitation training have promising rehabilitation outcomes. In this study, we examined whether participants could volitionally recall MI-related brain activation patterns when guided using neurofeedback (NF) during training. The participants’ performance was compared to that without NF. We hypothesized that participants would be able to consistently generate the relevant activation pattern associated with the MI task during training with NF compared to that without NF. To assess activation consistency, we used the performance of classifiers trained to discriminate MI-related brain activation patterns. Our results showed significantly higher predictive values of MI-related activation patterns during training with NF. Additionally, this improvement in the classification performance tends to be associated with the activation of middle temporal gyrus/inferior occipital gyrus, a region associated with visual motion processing, suggesting the importance of performance monitoring during MI task training. Taken together, these findings suggest that the efficacy of MI training, in terms of generating consistent brain activation patterns relevant to the task, can be enhanced by using NF as a mechanism to enable participants to volitionally recall task-related brain activation patterns.

Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain-computer interface: three-class classification of rest, right-, and left-hand motor execution

The aim of this work is to develop an effective brain-computer interface (BCI) method based on functional near-infrared spectroscopy (fNIRS). In order to improve the performance of the BCI system in terms of accuracy, the ability to discriminate features from input signals and proper classification are desired. Previous studies have mainly extracted features from the signal manually, but proper features need to be selected carefully. To avoid performance degradation caused by manual feature selection, we applied convolutional neural networks (CNNs) as the automatic feature extractor and classifier for fNIRS-based BCI. In this study, the hemodynamic responses evoked by performing rest, right-, and left-hand motor execution tasks were measured on eight healthy subjects to compare performances. Our CNN-based method provided improvements in classification accuracy over conventional methods employing the most commonly used features of mean, peak, slope, variance, kurtosis, and skewness, classified by support vector machine (SVM) and artificial neural network (ANN). Specifically, up to 6.49% and 3.33% improvement in classification accuracy was achieved by CNN compared with SVM and ANN, respectively.

Computational approaches to fMRI analysis

Analysis methods in cognitive neuroscience have not always matched the richness of fMRI data. Early methods focused on estimating neural activity within individual voxels or regions, averaged over trials or blocks and modeled separately in each participant. This approach mostly neglected the distributed nature of neural representations over voxels, the continuous dynamics of neural activity during tasks, the statistical benefits of performing joint inference over multiple participants and the value of using predictive models to constrain analysis. Several recent exploratory and theory-driven methods have begun to pursue these opportunities. These methods highlight the importance of computational techniques in fMRI analysis, especially machine learning, algorithmic optimization and parallel computing. Adoption of these techniques is enabling a new generation of experiments and analyses that could transform our understanding of some of the most complex-and distinctly human-signals in the brain: acts of cognition such as thoughts, intentions and memories.

Progress and roadblocks in the search for brain-based biomarkers of autism and attention-deficit/hyperactivity disorder

Children with neurodevelopmental disorders benefit most from early interventions and treatments. The development and validation of brain-based biomarkers to aid in objective diagnosis can facilitate this important clinical aim. The objective of this review is to provide an overview of current progress in the use of neuroimaging to identify brain-based biomarkers for autism spectrum disorder (ASD) and attention-deficit/hyperactivity disorder (ADHD), two prevalent neurodevelopmental disorders. We summarize empirical work that has laid the foundation for using neuroimaging to objectively quantify brain structure and function in ways that are beginning to be used in biomarker development, noting limitations of the data currently available. The most successful machine learning methods that have been developed and applied to date are discussed. Overall, there is increasing evidence that specific features (for example, functional connectivity, gray matter volume) of brain regions comprising the salience and default mode networks can be used to discriminate ASD from typical development. Brain regions contributing to successful discrimination of ADHD from typical development appear to be more widespread, however there is initial evidence that features derived from frontal and cerebellar regions are most informative for classification. The identification of brain-based biomarkers for ASD and ADHD could potentially assist in objective diagnosis, monitoring of treatment response and prediction of outcomes for children with these neurodevelopmental disorders. At present, however, the field has yet to identify reliable and reproducible biomarkers for these disorders, and must address issues related to clinical heterogeneity, methodological standardization and cross-site validation before further progress can be achieved.

Real-time fMRI data for testing OpenNFT functionality

Here, we briefly describe the real-time fMRI data that is provided for testing the functionality of the open-source Python/Matlab framework for neurofeedback, termed Open NeuroFeedback Training (OpenNFT, Koush et al. [1]). The data set contains real-time fMRI runs from three anonymized participants (i.e., one neurofeedback run per participant), their structural scans and pre-selected ROIs/masks/weights. The data allows for simulating the neurofeedback experiment without an MR scanner, exploring the software functionality, and measuring data processing times on the local hardware. In accordance with the descriptions in our main article, we provide data of (1) periodically displayed (intermittent) activation-based feedback; (2) intermittent effective connectivity feedback, based on dynamic causal modeling (DCM) estimations; and (3) continuous classification-based feedback based on support-vector-machine (SVM) estimations. The data is available on our public GitHub repository:

https://github.com/OpenNFT/OpenNFT_Demo/releases.

Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.

Real-time functional magnetic resonance imaging in obsessive-compulsive disorder

The current literature provides substantial evidence of brain alterations associated with obsessive-compulsive disorder (OCD) symptoms (eg, checking, cleaning/decontamination, counting compulsions; harm or sexual, symmetry/exactness obsessions), and emotional problems (eg, defensive/appetitive emotional imbalance, disgust, guilt, shame, and fear learning/extinction) and cognitive impairments associated with this disorder (eg, inhibitory control, working memory, cognitive flexibility). Building on this evidence, new clinical trials can now target specific brain regions/networks. Real-time functional magnetic resonance imaging (rtfMRI) was introduced as a new therapeutic tool for the self-regulation of brain-mind. In this review, we describe initial trials testing the use of rtfMRI to target brain regions associated with specific OCD symptoms (eg, contamination), and other mind-brain processes (eg, cognitive - working memory, inhibitory control, emotional - defensive, appetitive systems, fear reduction through counter-conditioning) found impaired in OCD patients. While this is a novel topic of research, initial evidence shows the promise of using rtfMRI in training the self-regulation of brain regions and mental processes associated with OCD. Additionally, studies with healthy populations have shown that individuals can regulate brain regions associated with cognitive and emotional processes found impaired in OCD. After the initial "proof-of-concept" stage, there is a need to follow up with controlled clinical trials that could test rtfMRI innovative treatments targeting brain regions and networks associated with different OCD symptoms and cognitive-emotional impairments.

OpenNFT: An open-source Python/Matlab framework for real-time fMRI neurofeedback training based on activity, connectivity and multivariate pattern analysis

Neurofeedback based on real-time functional magnetic resonance imaging (rt-fMRI) is a novel and rapidly developing research field. It allows for training of voluntary control over localized brain activity and connectivity and has demonstrated promising clinical applications. Because of the rapid technical developments of MRI techniques and the availability of high-performance computing, new methodological advances in rt-fMRI neurofeedback become possible. Here we outline the core components of a novel open-source neurofeedback framework, termed Open NeuroFeedback Training (OpenNFT), which efficiently integrates these new developments. This framework is implemented using Python and Matlab source code to allow for diverse functionality, high modularity, and rapid extendibility of the software depending on the user's needs. In addition, it provides an easy interface to the functionality of Statistical Parametric Mapping (SPM) that is also open-source and one of the most widely used fMRI data analysis software. We demonstrate the functionality of our new framework by describing case studies that include neurofeedback protocols based on brain activity levels, effective connectivity models, and pattern classification approaches. This open-source initiative provides a suitable framework to actively engage in the development of novel neurofeedback approaches, so that local methodological developments can be easily made accessible to a wider range of users.

A network engineering perspective on probing and perturbing cognition with neurofeedback

Network science and engineering provide a flexible and generalizable tool set to describe and manipulate complex systems characterized by heterogeneous interaction patterns among component parts. While classically applied to social systems, these tools have recently proven to be particularly useful in the study of the brain. In this review, we describe the nascent use of these tools to understand human cognition, and we discuss their utility in informing the meaningful and predictable perturbation of cognition in combination with the emerging capabilities of neurofeedback. To blend these disparate strands of research, we build on emerging conceptualizations of how the brain functions (as a complex network) and how we can develop and target interventions or modulations (as a form of network control). We close with an outline of current frontiers that bridge neurofeedback, connectomics, and network control theory to better understand human cognition.

Optimization of functional MRI for detection, decoding and high-resolution imaging of the response patterns of cortical columns

The capacity of functional MRI (fMRI) to resolve cortical columns depends on several factors. These include the spatial scale of the columnar pattern, the point-spread of the fMRI response, the voxel size, and the signal-to-noise ratio (SNR) considering thermal and physiological noise. However, it remains unknown how these factors combine, and what is the voxel size that optimizes fMRI of cortical columns. Here we combine current knowledge into a quantitative model of fMRI of realistic patterns of cortical columns with different spatial scales and degrees of irregularity. We compare different approaches for identifying patterns of cortical columns, including univariate and multivariate based detection, multi-voxel pattern analysis (MVPA) based decoding, and high-resolution imaging and reconstruction of the pattern of cortical columns. We present the dependence of the performance of each approach on the parameters of the imaged pattern as well as those of the data acquisition. In addition, we predict voxel sizes that optimize fMRI of cortical columns under various scenarios. We found that all measures associated with multivariate detection and decoding could be approximately calculated from a measure we termed "multivariate contrast-to-noise ratio" (mv-CNR), which is a function of the contrast-to-noise ratio (CNR) and number of voxels. Furthermore, mv-CNR implied that the optimal voxel width for detection and decoding is independent of changes in response amplitude, SNR and imaged volume that are not caused by changes in voxel size. For regular patterns, optimal voxel widths for detection, decoding and imaging/reconstructing the pattern of cortical columns were approximately half the main cycle length of the organization. Optimal voxel widths for irregular patterns were less dependent on the main cycle length, and differed between univariate detection, multivariate detection and decoding, and reconstruction. We compared the effects of different factors of Gradient Echo fMRI at 3 Tesla (T), Gradient Echo fMRI at 7T, and Spin-Echo fMRI at 7T on the detection, decoding, and reconstruction measures considered and found that in all cases the width of the fMRI point-spread had the most significant effect. In contrast, different response amplitudes and noise characteristics played a relatively minor role. We recommend specific voxel widths for optimal univariate detection, for multivariate detection and decoding, and for high-resolution imaging of cortical columns under these three data-acquisition scenarios. Our study supports the planning, optimization, and interpretation of high-resolution fMRI of cortical columns and the decoding of information conveyed by these columns.

Decoded fMRI neurofeedback can induce bidirectional confidence changes within single participants

Neurofeedback studies using real-time functional magnetic resonance imaging (rt-fMRI) have recently incorporated the multi-voxel pattern decoding approach, allowing for fMRI to serve as a tool to manipulate fine-grained neural activity embedded in voxel patterns. Because of its tremendous potential for clinical applications, certain questions regarding decoded neurofeedback (DecNef) must be addressed. Specifically, can the same participants learn to induce neural patterns in opposite directions in different sessions? If so, how does previous learning affect subsequent induction effectiveness? These questions are critical because neurofeedback effects can last for months, but the short- to mid-term dynamics of such effects are unknown. Here we employed a within-subjects design, where participants underwent two DecNef training sessions to induce behavioural changes of opposing directionality (up or down regulation of perceptual confidence in a visual discrimination task), with the order of training counterbalanced across participants. Behavioral results indicated that the manipulation was strongly influenced by the order and the directionality of neurofeedback training. We applied nonlinear mathematical modeling to parametrize four main consequences of DecNef: main effect of change in confidence, strength of down-regulation of confidence relative to up-regulation, maintenance of learning effects, and anterograde learning interference. Modeling results revealed that DecNef successfully induced bidirectional confidence changes in different sessions within single participants. Furthermore, the effect of up- compared to down-regulation was more prominent, and confidence changes (regardless of the direction) were largely preserved even after a week-long interval. Lastly, the effect of the second session was markedly diminished as compared to the effect of the first session, indicating strong anterograde learning interference. These results are interpreted in the framework of reinforcement learning and provide important implications for its application to basic neuroscience, to occupational and sports training, and to therapy.