Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Deep multimodal representations and classification of first-episode psychosis via live face processing

Schizophrenia is a severe psychiatric disorder associated with a wide range of cognitive and neurophysiological dysfunctions and long-term social difficulties. Early detection is expected to reduce the burden of disease by initiating early treatment. In this paper, we test the hypothesis that integration of multiple simultaneous acquisitions of neuroimaging, behavioral, and clinical information will be better for prediction of early psychosis than unimodal recordings. We propose a novel framework to investigate the neural underpinnings of the early psychosis symptoms (that can develop into Schizophrenia with age) using multimodal acquisitions of neural and behavioral recordings including functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), and facial features. Our data acquisition paradigm is based on live face-toface interaction in order to study the neural correlates of social cognition in first-episode psychosis (FEP). We propose a novel deep representation learning framework, Neural-PRISM, for learning joint multimodal compressed representations combining neural as well as behavioral recordings. These learned representations are subsequently used to describe, classify, and predict the severity of early psychosis in patients, as measured by the Positive and Negative Syndrome Scale (PANSS) and Global Assessment of Functioning (GAF) scores to evaluate the impact of symptomatology. We found that incorporating joint multimodal representations from fNIRS and EEG along with behavioral recordings enhances classification between typical controls and FEP individuals (significant improvements between 10 - 20%). Additionally, our results suggest that geometric and topological features such as curvatures and path signatures of the embedded trajectories of brain activity enable detection of discriminatory neural characteristics in early psychosis.

Epileptic brain network mechanisms and neuroimaging techniques for the brain network

Epilepsy can be defined as a dysfunction of the brain network, and each type of epilepsy involves different brain-network changes that are implicated differently in the control and propagation of interictal or ictal discharges. Gaining more detailed information on brain network alterations can help us to further understand the mechanisms of epilepsy and pave the way for brain network-based precise therapeutic approaches in clinical practice. An increasing number of advanced neuroimaging techniques and electrophysiological techniques such as diffusion tensor imaging-based fiber tractography, diffusion kurtosis imaging-based fiber tractography, fiber ball imaging-based tractography, electroencephalography, functional magnetic resonance imaging, magnetoencephalography, positron emission tomography, molecular imaging, and functional ultrasound imaging have been extensively used to delineate epileptic networks. In this review, we summarize the relevant neuroimaging and neuroelectrophysiological techniques for assessing structural and functional brain networks in patients with epilepsy, and extensively analyze the imaging mechanisms, advantages, limitations, and clinical application ranges of each technique. A greater focus on emerging advanced technologies, new data analysis software, a combination of multiple techniques, and the construction of personalized virtual epilepsy models can provide a theoretical basis to better understand the brain network mechanisms of epilepsy and make surgical decisions.

Deep Multimodal Representations and Classification of First-Episode Psychosis via Live Face Processing

Schizophrenia is a severe psychiatric disorder associated with a wide range of cognitive and neurophysiological dysfunctions and long-term social difficulties. In this paper, we test the hypothesis that integration of multiple simultaneous acquisitions of neuroimaging, behavioral, and clinical information will be better for prediction of early psychosis than unimodal recordings. We propose a novel framework to investigate the neural underpinnings of the early psychosis symptoms (that can develop into Schizophrenia with age) using multimodal acquisitions of neural and behavioral recordings including functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), and facial features. Our data acquisition paradigm is based on live face-to-face interaction in order to study the neural correlates of social cognition in first-episode psychosis (FEP). We propose a novel deep representation learning framework, Neural-PRISM, for learning joint multimodal compressed representations combining neural as well as behavioral recordings. These learned representations are subsequently used to describe, classify, and predict the severity of early psychosis in patients, as measured by the Positive and Negative Syndrome Scale (PANSS) and Global Assessment of Functioning (GAF) scores. We found that incorporating joint multimodal representations from fNIRS and EEG along with behavioral recordings enhances classification between typical controls and FEP individuals. Additionally, our results suggest that geometric and topological features such as curvatures and path signatures of the embedded trajectories of brain activity enable detection of discriminatory neural characteristics in early psychosis.

Beta-Band Cortico-Muscular Phase Coherence in Hemiparetic Stroke

Following a stroke, compensation for the loss of ipsilesional corticospinal and corticobulbar projections, results in increased reliance on contralesional motor pathways during paretic arm movement. Better understanding outcomes of post-stroke contralesional cortical adaptation outcomes may benefit more targeted post-stroke motor rehabilitation interventions. This proof-of-concept study involves eight healthy controls and ten post-stroke participants. Electroencephalographic (EEG) and deltoid electromyographic (EMG) data were collected during an upper-limb task. Phase coupling between beta-band motor cortex EEG and deltoid EMG was assessed using the Multi-Phase Locking Value (M-PLV) method. Different from classic cortico-muscular coherence, M-PLV allows for the calculation of dynamic phase coherence and delays, and is not affected by the non-stationary nature of EEG/EMG signals. Nerve conduction delay from the contralateral motor cortex to the deltoid muscle of the paretic arm was estimated. Our results show the ipsilateral (contralesional) motor cortex beta-band phase coherence behavior is altered in stroke participants, with significant differences in ipsilateral EEG-EMG coherence values, ipsilateral time course percentage above the significance threshold, and ipsilateral time course area above the significance threshold. M-PLV phase coherence analysis provides evidence for post-stroke contralesional motor adaptation, highlighting its increased role in the paretic shoulder abduction task. Nerve conduction delay between the motor cortices and deltoid muscle is significantly higher in stroke participants. Beta-band M-PLV phase coherence analysis shows greater phase-coherence distribution convergence between the ipsilateral (contralesional) and contralateral (ipsilesional) motor cortices in stroke participants, which is interpretable as evidence of maladaptive neural adaptation resulting from a greater reliance on the contralesional motor cortices.

NiReject: toward automated bad channel detection in functional near-infrared spectroscopy

Significance

The increasing sample sizes and channel densities in functional near-infrared spectroscopy (fNIRS) necessitate precise and scalable identification of signals that do not permit reliable analysis to exclude them. Despite the relevance of detecting these "bad channels," little is known about the behavior of fNIRS detection methods, and the potential of unsupervised and semi-supervised machine learning remains unexplored.

Aim

We developed three novel machine learning-based detectors, unsupervised, semi-supervised, and hybrid NiReject, and compared them with existing approaches.

Approach

We conducted a systematic literature search and demonstrated the influence of bad channel detection. Based on 29,924 signals from two independently rated datasets and a simulated scenario space of diverse phenomena, we evaluated the NiReject models, six of the most established detection methods in fNIRS, and 11 prominent methods from other domains.

Results

Although the results indicated that a lack of proper detection can strongly bias findings, detection methods were reported in only 32% of the included studies. Semi-supervised models, specifically semi-supervised NiReject, outperformed both established thresholding-based and unsupervised detectors. Hybrid NiReject, utilizing a human feedback loop, addressed the practical challenges of semi-supervised methods while maintaining precise detection and low rating effort.

Conclusions

This work contributes toward more automated and reliable fNIRS signal quality control by comprehensively evaluating existing and introducing novel machine learning-based techniques and outlining practical considerations for bad channel detection.

Functional Near-Infrared Spectroscopy as a Personalized Digital Healthcare Tool for Brain Monitoring

The sustained growth of digital healthcare in the field of neurology relies on portable and cost-effective brain monitoring tools that can accurately monitor brain function in real time. Functional near-infrared spectroscopy (fNIRS) is one such tool that has become popular among researchers and clinicians as a practical alternative to functional magnetic resonance imaging, and as a complementary tool to modalities such as electroencephalography. This review covers the contribution of fNIRS to the personalized goals of digital healthcare in neurology by identifying two major trends that drive current fNIRS research. The first major trend is multimodal monitoring using fNIRS, which allows clinicians to access more data that will help them to understand the interconnection between the cerebral hemodynamics and other physiological phenomena in patients. This allows clinicians to make an overall assessment of physical health to obtain a more-detailed and individualized diagnosis. The second major trend is that fNIRS research is being conducted with naturalistic experimental paradigms that involve multisensory stimulation in familiar settings. Cerebral monitoring of multisensory stimulation during dynamic activities or within virtual reality helps to understand the complex brain activities that occur in everyday life. Finally, the scope of future fNIRS studies is discussed to facilitate more-accurate assessments of brain activation and the wider clinical acceptance of fNIRS as a medical device for digital healthcare.

Error-related brain state analysis using electroencephalography in conjunction with functional near-infrared spectroscopy during a complex surgical motor task

Error-based learning is one of the basic skill acquisition mechanisms that can be modeled as a perception-action system and investigated based on brain-behavior analysis during skill training. Here, the error-related chain of mental processes is postulated to depend on the skill level leading to a difference in the contextual switching of the brain states on error commission. Therefore, the objective of this paper was to compare error-related brain states, measured with multi-modal portable brain imaging, between experts and novices during the Fundamentals of Laparoscopic Surgery (FLS) "suturing and intracorporeal knot-tying" task (FLS complex task)-the most difficult among the five psychomotor FLS tasks. The multi-modal portable brain imaging combined functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) for brain-behavior analysis in thirteen right-handed novice medical students and nine expert surgeons. The brain state changes were defined by quasi-stable EEG scalp topography (called microstates) changes using 32-channel EEG data acquired at 250 Hz. Six microstate prototypes were identified from the combined EEG data from experts and novices during the FLS complex task that explained 77.14% of the global variance. Analysis of variance (ANOVA) found that the proportion of the total time spent in different microstates during the 10-s error epoch was significantly affected by the skill level (p < 0.01), the microstate type (p < 0.01), and the interaction between the skill level and the microstate type (p < 0.01). Brain activation based on the slower oxyhemoglobin (HbO) changes corresponding to the EEG band power (1-40 Hz) changes were found using the regularized temporally embedded Canonical Correlation Analysis of the simultaneously acquired fNIRS-EEG signals. The HbO signal from the overlying the left inferior frontal gyrus-opercular part, left superior frontal gyrus-medial orbital, left postcentral gyrus, left superior temporal gyrus, right superior frontal gyrus-medial orbital cortical areas showed significant (p < 0.05) difference between experts and novices in the 10-s error epoch. We conclude that the difference in the error-related chain of mental processes was the activation of cognitive top-down attention-related brain areas, including left dorsolateral prefrontal/frontal eye field and left frontopolar brain regions, along with a 'focusing' effect of global suppression of hemodynamic activation in the experts, while the novices had a widespread stimulus(error)-driven hemodynamic activation without the 'focusing' effect.

Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic, Methodology-Focused Review

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) stand as state-of-the-art techniques for non-invasive functional neuroimaging. On a unimodal basis, EEG has poor spatial resolution while presenting high temporal resolution. In contrast, fNIRS offers better spatial resolution, though it is constrained by its poor temporal resolution. One important merit shared by the EEG and fNIRS is that both modalities have favorable portability and could be integrated into a compatible experimental setup, providing a compelling ground for the development of a multimodal fNIRS-EEG integration analysis approach. Despite a growing number of studies using concurrent fNIRS-EEG designs reported in recent years, the methodological reference of past studies remains unclear. To fill this knowledge gap, this review critically summarizes the status of analysis methods currently used in concurrent fNIRS-EEG studies, providing an up-to-date overview and guideline for future projects to conduct concurrent fNIRS-EEG studies. A literature search was conducted using PubMed and Web of Science through 31 August 2021. After screening and qualification assessment, 92 studies involving concurrent fNIRS-EEG data recordings and analyses were included in the final methodological review. Specifically, three methodological categories of concurrent fNIRS-EEG data analyses, including EEG-informed fNIRS analyses, fNIRS-informed EEG analyses, and parallel fNIRS-EEG analyses, were identified and explained with detailed description. Finally, we highlighted current challenges and potential directions in concurrent fNIRS-EEG data analyses in future research.

Error related fNIRS-EEG microstate analysis during a complex surgical motor task

Fundamentals of Laparoscopic Surgery (FLS) is a standard education and training module with a set of basic surgical skills. During surgical skill acquisition, novices need to learn from errors due to perturbations in their performance which is one of the basic principles of motor skill acquisition. This study on thirteen healthy novice medical students and nine expert surgeons aimed to capture the brain state during error epochs using multimodal brain imaging by combining functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG). We performed error-related microstate analysis in the latent space that was found using regularized temporally embedded Canonical Correlation Analysis from fNIRS-EEG recordings during the performance of FLS "suturing and intracorporeal knot-tying" task - the most difficult among the five psychomotor FLS tasks. We found from two-way analysis of variance (ANDVA) with factors, skill level (expert, novice), and microstate type (1-6) that the proportion of the total time spent in microstates in the error epochs was significantly affected by the skill level ( ), microstate type ( ), and the interaction between the skill level and the microstate type ( ). Therefore, our study highlighted the relevance of portable brain imaging to capture error behavior when comparing the skill level during a complex surgical task. Clinical Relevance-This establishes the brain-behavior relationship for monitoring complex surgical motor task errors that differentiated experts from novices.