Multimodal Depression Detection: Fusion of Electroencephalography and Paralinguistic Behaviors Using a Novel Strategy for Classifier Ensemble

Depression assessment using integrated multi-featured EEG bands deep neural network models: Leveraging ensemble learning techniques

Mental Status Assessment (MSA) holds significant importance in psychiatry. In recent years, several studies have leveraged Electroencephalogram (EEG) technology to gauge an individual's mental state or level of depression. This study introduces a novel multi-tier ensemble learning approach to integrate multiple EEG bands for conducting mental state or depression assessments. Initially, the EEG signal is divided into eight sub-bands, and then a Long Short-Term Memory (LSTM)-based Deep Neural Network (DNN) model is trained for each band. Subsequently, the integration of multi-band EEG frequency models and the evaluation of mental state or depression level are facilitated through a two-tier ensemble learning approach based on Multiple Linear Regression (MLR). The authors conducted numerous experiments to validate the performance of the proposed method under different evaluation metrics. For clarity and conciseness, the research employs the simplest commercialized one-channel EEG sensor, positioned at FP1, to collect data from 57 subjects (49 depressed and 18 healthy subjects). The obtained results, including an accuracy of 0.897, F1-score of 0.921, precision of 0.935, negative predictive value of 0.829, recall of 0.908, specificity of 0.875, and AUC of 0.8917, provide evidence of the superior performance of the proposed method compared to other ensemble learning techniques. This method not only proves effective but also holds the potential to significantly enhance the accuracy of depression assessment.

Multimodal Mental Health Digital Biomarker Analysis From Remote Interviews Using Facial, Vocal, Linguistic, and Cardiovascular Patterns

OBJECTIVE

Psychiatric evaluation suffers from subjectivity and bias, and is hard to scale due to intensive professional training requirements. In this work, we investigated whether behavioral and physiological signals, extracted from tele-video interviews, differ in individuals with psychiatric disorders.

METHODS

Temporal variations in facial expression, vocal expression, linguistic expression, and cardiovascular modulation were extracted from simultaneously recorded audio and video of remote interviews. Averages, standard deviations, and Markovian process-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination.

RESULTS

Statistically significant feature differences were found between psychiatric and control subjects. Correlations were found between features and self-rated depression and anxiety scores. Heart rate dynamics provided the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities provided AUROCs of 0.72-0.82.

CONCLUSION

Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status.

SIGNIFICANCE

The proposed multimodal approach has the potential to facilitate scalable, remote, and low-cost assessment for low-burden automated mental health services.

Resting-state EEG dynamic functional connectivity distinguishes non-psychotic major depression, psychotic major depression and schizophrenia

This study aims to identify dynamic patterns within the spatiotemporal feature space that are specific to nonpsychotic major depression (NPMD), psychotic major depression (PMD), and schizophrenia (SCZ). The study also evaluates the effectiveness of machine learning algorithms based on these network manifestations in differentiating individuals with NPMD, PMD, and SCZ. A total of 579 participants were recruited, including 152 patients with NPMD, 45 patients with PMD, 185 patients with SCZ, and 197 healthy controls (HCs). A dynamic functional connectivity (DFC) approach was employed to estimate the principal FC states within each diagnostic group. Incremental proportions of data (ranging from 10% to 100%) within each diagnostic group were used for variability testing. DFC metrics, such as proportion, mean duration, and transition number, were examined among the four diagnostic groups to identify disease-related neural activity patterns. These patterns were then used to train a two-layer classifier for the four groups (HC, NPMD, PMD, and SCZ). The four principal brain states (i.e., states 1,2,3, and 4) identified by the DFC approach were highly representative within and across diagnostic groups. Between-group comparisons revealed significant differences in network metrics of state 2 and state 3, within delta, theta, and gamma frequency bands, between healthy individuals and patients in each diagnostic group (p < 0.01, FDR corrected). Moreover, the identified key dynamic network metrics achieved an accuracy of 73.1 ± 2.8% in the four-way classification of HC, NPMD, PMD, and SCZ, outperforming the static functional connectivity (SFC) approach (p < 0.001). These findings suggest that the proposed DFC approach can identify dynamic network biomarkers at the single-subject level. These biomarkers have the potential to accurately differentiate individual subjects among various diagnostic groups of psychiatric disorders or healthy controls. This work may contribute to the development of a valuable EEG-based diagnostic tool with enhanced accuracy and assistive capabilities.

Detecting depression based on facial cues elicited by emotional stimuli in video

Recently, depression research has received considerable attention and there is an urgent need for objective and validated methods to detect depression. Depression detection based on facial expressions may be a promising adjunct to depression detection due to its non-contact nature. Stimulated facial expressions may contain more information that is useful in detecting depression than natural facial expressions. To explore facial cues in healthy controls and depressed patients in response to different emotional stimuli, facial expressions of 62 subjects were collected while watching video stimuli, and a local face reorganization method for depression detection is proposed. The method extracts the local phase pattern features, facial action unit (AU) features and head motion features of a local face reconstructed according to facial proportions, and then fed into the classifier for classification. The classification accuracy was 76.25%, with a recall of 80.44% and a specificity of 83.21%. The results demonstrated that the negative video stimuli in the single-attribute stimulus analysis were more effective in eliciting changes in facial expressions in both healthy controls and depressed patients. Fusion of facial features under both neutral and negative stimuli was found to be useful in discriminating between healthy controls and depressed individuals. The Pearson correlation coefficient (PCC) showed that changes in the emotional stimulus paradigm were more strongly correlated with changes in subjects' facial AU when exposed to negative stimuli compared to stimuli of other attributes. These results demonstrate the feasibility of our proposed method and provide a framework for future work in assisting diagnosis.

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

Objective

The current clinical practice of psychiatric evaluation suffers from subjectivity and bias, and requires highly skilled professionals that are often unavailable or unaffordable. Objective digital biomarkers have shown the potential to address these issues. In this work, we investigated whether behavioral and physiological signals, extracted from remote interviews, provided complimentary information for assessing psychiatric disorders.

Methods

Time series of multimodal features were derived from four conceptual modes: facial expression, vocal expression, linguistic expression, and cardiovascular modulation. The features were extracted from simultaneously recorded audio and video of remote interviews using task-specific and foundation models. Averages, standard deviations, and hidden Markov model-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination.

Results

Statistically significant feature differences were found between controls and subjects with mental health conditions. Correlations were found between features and self-rated depression and anxiety scores. Visual heart rate dynamics achieved the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities achieved AUROCs of 0.72-0.82. Features from task-specific models outperformed features from foundation models.

Conclusion

Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status.

Significance

The proposed multimodal approach has the potential to facilitate objective, remote, and low-cost assessment for low-burden automated mental health services.

What Can Facial Movements Reveal? Depression Recognition and Analysis Based on Optical Flow Using Bayesian Networks

Recent evidence have demonstrated that facial expressions could be a valid and important aspect for depression recognition. Although various works have been achieved in automatic depression recognition, it is a challenge to explore the inherent nuances of facial expressions that might reveal the underlying differences between depressed patients and healthy subjects under different stimuli. There is a lack of an undisturbed system that monitors depressive patients' mental states in various free-living scenarios, so this paper steps towards building a classification model where data collection, feature extraction, depression recognition and facial actions analysis are conducted to infer the differences of facial movements between depressive patients and healthy subjects. In this study, we firstly present a plan of dividing facial regions of interest to extract optical flow features of facial expressions for depression recognition. We then propose facial movements coefficients utilising discrete wavelet transformation. Specifically, Bayesian Networks equipped with construction of Pearson Correlation Coefficients based on discrete wavelet transformation is learnt, which allows for analysing movements of different facial regions. We evaluate our method on a clinically validated dataset of 30 depressed patients and 30 healthy control subjects, and experiments results obtained the accuracy and recall of 81.7%, 96.7%, respectively, outperforming other features for comparison. Most importantly, the Bayesian Networks we built on the coefficients under different stimuli may reveal some facial action patterns of depressed subjects, which have a potential to assist the automatic diagnosis of depression.

Cross-subject classification of depression by using multiparadigm EEG feature fusion

BACKGROUND AND OBJECTIVE

The aim of this study is to solve the non-stationarity and complexity characteristics and huge individual differences in the electroencephalogram (EEG) signals for depression classification.

METHODS

To address those problems, the Lempel-Ziv complexity feature matrices were extracted from the EEG signals under the two paradigms of eyes open and eyes closed in the resting state. Topographical map of brain and statistical analysis were introduced to investigate the significance of eyes open and eyes closed EEG for depression classification. To promote the classification accuracy, feature matrices from the two paradigms were fused. And linear combination and concatenation fusion methods were proposed to further reveal the underlying mechanism of improving classification accuracy. Support vector machine (SVM), K-nearest neighbor, and decision tree classifiers were employed and compared to classify depression under the eyes open, eyes closed and fused paradigm.

RESULTS

The classification results of 10-fold cross-validation showed that the highest average accuracy (86.58%) under a single paradigm was achieved in the eyes-open state. The multiparadigm fusion method of concatenation was better than the linear combination. The best classification result was obtained using multiparadigm feature concatenation under the SVM classifier, yielding an accuracy of 94.03%.

CONCLUSION

The multiparadigm feature fusion method proposed in this paper can effectively improve the accuracy of depression classification. It was proved that eyes open and eyes closed EEG have complementary information, which was benefit for the cross-subject classification of depression. It provides new ideas for depression classification in clinics.

Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network

Motor imagery-based brain-computer interfaces (MI-BCI) have important application values in the field of neurorehabilitation and robot control. At present, MI-BCI mostly use bilateral upper limb motor tasks, but there are relatively few studies on single upper limb MI tasks. In this work, we conducted studies on the recognition of motor imagery EEG signals of the right upper limb and proposed a multi-branch fusion convolutional neural network (MF-CNN) for learning the features of the raw EEG signals as well as the two-dimensional time-frequency maps at the same time. The dataset used in this study contained three types of motor imagery tasks: extending the arm, rotating the wrist, and grasping the object, 25 subjects were included. In the binary classification experiment between the grasping object and the arm-extending tasks, MF-CNN achieved an average classification accuracy of 78.52% and kappa value of 0.57. When all three tasks were used for classification, the accuracy and kappa value were 57.06% and 0.36, respectively. The comparison results showed that the classification performance of MF-CNN is higher than that of single CNN branch algorithms in both binary-class and three-class classification. In conclusion, MF-CNN makes full use of the time-domain and frequency-domain features of EEG, can improve the decoding accuracy of single limb motor imagery tasks, and it contributes to the application of MI-BCI in motor function rehabilitation training after stroke.

Feature-level fusion based on spatial-temporal of pervasive EEG for depression recognition

BACKGROUND AND OBJECTIVE

In view of the depression characteristics such as high prevalence, high disability rate, high fatality rate, and high recurrence rate, early identification and early intervention are the most effective methods to prevent irreversible damage of brain function over time. The traditional method of depression recognition based on questionnaires and interviews is time-consuming and labor-intensive, and heavily depends on the doctor's subjective experience. Therefore, accurate, convenient and effective recognition of depression has important social value and scientific significance.

METHODS

This paper proposes a depression recognition framework based on feature-level fusion of spatial-temporal pervasive electroencephalography (EEG). Time series EEG data were collected by portable three-electrode EEG acquisition instrument, and mapped to a spatial complex network called visibility graph (VG). Then temporal EEG features and spatial VG metric features were extracted and selected. Based on the correlation between features and categories, the differences in contribution of individual feature are explored, and different contribution coefficients are assigned to different features as the data basis of feature-level fusion to ensure the diversity of data. A cascade forest model based on three different decision forests is designed to realize the efficient depression recognition using spatial-temporal feature-level fusion data.

RESULTS

Experimental data were obtained from 26 depressed patients and 29 healthy controls (HC). The results of multiple control experiments show that compared with single type feature, feature-level fusion without contribution coefficient, and independent classifiers, the feature-level method with contribution coefficient of spatial-temporal has a stronger recognition ability of depression, and the highest accuracy is 92.48%.

CONCLUSION

Feature-level fusion method provides an effective computer-aided tool for rapid clinical diagnosis of depression.

High-order brain functional network for electroencephalography-based diagnosis of major depressive disorder

Brain functional network (BFN) based on electroencephalography (EEG) has been widely used to diagnose brain diseases, such as major depressive disorder (MDD). However, most existing BFNs only consider the correlation between two channels, ignoring the high-level interaction among multiple channels that contain more rich information for diagnosing brain diseases. In such a sense, the BFN is called low-order BFN (LO-BFN). In order to fully explore the high-level interactive information among multiple channels of the EEG signals, a scheme for constructing a high-order BFN (HO-BFN) based on the “correlation’s correlation” strategy is proposed in this paper. Specifically, the entire EEG time series is firstly divided into multiple epochs by sliding window. For each epoch, the short-term correlation between channels is calculated to construct a LO-BFN. The correlation time series of all channel pairs are formulated by these LO-BFNs obtained from all epochs to describe the dynamic change of short-term correlation along the time. To construct HO-BFN, we cluster all correlation time series to avoid the problems caused by high dimensionality, and the correlation of the average correlation time series from different clusters is calculated to reflect the high-order correlation among multiple channels. Experimental results demonstrate the efficiency of the proposed HO-BFN in MDD identification, and its integration with the LO-BFN can further improve the recognition rate.

An End-to-End Depression Recognition Method Based on EEGNet

Major depressive disorder (MDD) is a common and highly debilitating condition that threatens the health of millions of people. However, current diagnosis of depression relies on questionnaires that are highly correlated with physician experience and hence not completely objective. Electroencephalography (EEG) signals combined with deep learning techniques may be an objective approach to effective diagnosis of MDD. This study proposes an end-to-end deep learning framework for MDD diagnosis based on EEG signals. We used EEG signals from 29 healthy subjects and 24 patients with severe depression to calculate Accuracy, Precision, Recall, F1-Score, and Kappa coefficient, which were 90.98%, 91.27%, 90.59%, and 81.68%, respectively. In addition, we found that these values were highest when happy-neutral face pairs were used as stimuli for detecting depression. Compared with exiting methods for EEG-based MDD classification, ours can maintain stable model performance without re-calibration. The present results suggest that the method is highly accurate for diagnosis of MDD and can be used to develop an automatic plug-and-play EEG-based system for diagnosing depression.

A novel ensemble learning method using multiple objective particle swarm optimization for subject-independent EEG-based emotion recognition

Emotion recognition is a vital but challenging step in creating passive brain-computer interface applications. In recent years, many studies on electroencephalogram (EEG)-based emotion recognition have been conducted. Ensemble learning has been widely used in emotion recognition because of its superior accuracy and generalization. In this study, we proposed a novel ensemble learning method based on multiple objective particle swarm optimization for subject-independent EEG-based emotion recognition. First, we used a 4 s sliding time window with a 2 s overlap to extract 13 different features from EEG signals and construct a feature vector. Then, we employed L1 regularization to select effective features. Second, a model selection method was applied to choose the optimal basic analysis submodels. Afterward, we proposed an ensemble operator that converts the classification results of a single model from discrete values to continuous values to better characterize the classification results. Subsequently, multiple objective particle swarm optimization was adopted to confirm the optimal parameters of the ensemble learning model. Finally, we conducted extensive experiments on two public datasets: DEAP and SEED. Considering the generalization of the model, we applied leave-one-subject-out cross-validation to evaluate the performance of the model. The experimental results demonstrate that the proposed method achieves a better recognition performance than single methods, commonly used ensemble learning methods, and state-of-the-art methods. The average accuracies for arousal and valence are 65.70% and 64.22%, respectively, on the DEAP database, and the average accuracy on the SEED database is 84.44%.

An Optimal Channel Selection for EEG-Based Depression Detection via Kernel-Target Alignment

Depression is a mental disorder with emotional and cognitive dysfunction. The main clinical characteristic of depression is significant and persistent low mood. As reported, depression is a leading cause of disability worldwide. Moreover, the rate of recognition and treatment for depression is low. Therefore, the detection and treatment of depression are urgent. Multichannel electroencephalogram (EEG) signals, which reflect the working status of the human brain, can be used to develop an objective and promising tool for augmenting the clinical effects in the diagnosis and detection of depression. However, when a large number of EEG channels are acquired, the information redundancy and computational complexity of the EEG signals increase; thus, effective channel selection algorithms are required not only for machine learning feasibility, but also for practicality in clinical depression detection. Consequently, we propose an optimal channel selection method for EEG-based depression detection via kernel-target alignment (KTA) to effectively resolve the abovementioned issues. In this method, we consider a modified version KTA that can measure the similarity between the kernel matrix for channel selection and the target matrix as an objective function and optimize the objective function by a proposed optimal channel selection strategy. Experimental results on two EEG datasets show that channel selection can effectively increase the classification performance and that even if we rely only on a small subset of channels, the results are still acceptable. The selected channels are in line with the expected latent cortical activity patterns in depression detection. Moreover, the experimental results demonstrate that our method outperforms the state-of-the-art channel selection approaches.

Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation

Self-Rating Depression Scale (SDS) questionnaire has frequently been used for efficient depression preliminary screening. However, the uncontrollable self-administered measure can be easily affected by insouciantly or deceptively answering, and producing the different results with the clinician-administered Hamilton Depression Rating Scale (HDRS) and the final diagnosis. Clinically, facial expression (FE) and actions play a vital role in clinician-administered evaluation, while FE and action are underexplored for self-administered evaluations. In this work, we collect a novel dataset of 200 subjects to evidence the validity of self-rating questionnaires with their corresponding question-wise video recording. To automatically interpret depression from the SDS evaluation and the paired video, we propose an end-to-end hierarchical framework for the long-term variable-length video, which is also conditioned on the questionnaire results and the answering time. Specifically, we resort to a hierarchical model which utilizes a 3D CNN for local temporal pattern exploration and a redundancy-aware self-attention (RAS) scheme for question-wise global feature aggregation. Targeting for the redundant long-term FE video processing, our RAS is able to effectively exploit the correlations of each video clip within a question set to emphasize the discriminative information and eliminate the redundancy based on feature pair-wise affinity. Then, the question-wise video feature is concatenated with the questionnaire scores for final depression detection. Our thorough evaluations also show the validity of fusing SDS evaluation and its video recording, and the superiority of our framework to the conventional state-of-the-art temporal modeling methods.

Brain Functional Networks Based on Resting-State EEG Data for Major Depressive Disorder Analysis and Classification

If the brain is regarded as a system, it will be one of the most complex systems in the universe. Traditional analysis and classification methods of major depressive disorder (MDD) based on electroencephalography (EEG) feature-levels often regard electrode as isolated node and ignore the correlation between them, so it's difficult to find alters of abnormal topological architecture in brain. To solve this problem, we propose a brain functional network framework for MDD of analysis and classification based on resting state EEG. The phase lag index (PLI) was calculated based on the 64-channel resting state EEG to construct the function connection matrix to reduce and avoid the volume conductor effect. Then binarization of brain function network based on small world index was realized. Statistical analyses were performed on different EEG frequency band and different brain regions. The results showed that significant alterations of brain synchronization occurred in frontal, temporal, parietal-occipital regions of left brain and temporal region of right brain. And average shortest path length and clustering coefficient in left central region of theta band and node betweenness centrality in right parietal-occipital region were significantly correlated with PHQ-9 score of MDD, which indicates these three network metrics may be served as potential biomarkers to effectively distinguish MDD from controls and the highest classification accuracy can reach 93.31%. Our findings also point out that the brain function network of MDD patients shows a random trend, and small world characteristics appears to weaken.