Using speech recognition technology to investigate the association between timing-related speech features and depression severity

Emotion-dependent language featuring depression

BACKGROUND AND OBJECTIVES

Understanding language features of depression contributes to the detection of the disorder. Considering that depression is characterized by dysfunctions in emotion and individuals with depression often show emotion-dependent cognition, the present study investigated the speech features and word use of emotion-dependent narrations in patients with depression.

METHODS

Forty depression patients and forty controls were required to narrate self-relevant memories under five basic human emotions (i.e., sad, angry, fearful, neutral, and happy). Recorded speech and transcribed texts were analyzed.

RESULTS

Patients with depression, as compared to non-depressed individuals, talked slower and less. They also performed differently in using negative emotion, work, family, sex, biology, health, and assent words regardless of emotion manipulation. Moreover, the use of words such as first person singular pronoun, past tense, causation, achievement, family, death, psychology, impersonal pronoun, quantifier and preposition words displayed emotion-dependent differences between groups. With the involvement of emotion, linguistic indicators associated with depressive symptoms were identified and explained 71.6% variances of depression severity.

LIMITATIONS

Word use was analyzed based on the dictionary which does not cover all the words spoken in the memory task, resulting in text data loss. Besides, a relatively small number of depression patients were included in the present study and therefore the results need confirmation in future research using big emotion-dependent data of speech and texts.

CONCLUSIONS

Our findings suggest that consideration of different emotional contexts is an effective means to improve the accuracy of depression detection via the analysis of word use and speech features.

Multilingual markers of depression in remotely collected speech samples: A preliminary analysis

BACKGROUND

Speech contains neuromuscular, physiological and cognitive components, and so is a potential biomarker of mental disorders. Previous studies indicate that speaking rate and pausing are associated with major depressive disorder (MDD). However, results are inconclusive as many studies are small and underpowered and do not include clinical samples. These studies have also been unilingual and use speech collected in controlled settings. If speech markers are to help understand the onset and progress of MDD, we need to uncover markers that are robust to language and establish the strength of associations in real-world data.

METHODS

We collected speech data in 585 participants with a history of MDD in the United Kingdom, Spain, and Netherlands as part of the RADAR-MDD study. Participants recorded their speech via smartphones every two weeks for 18 months. Linear mixed models were used to estimate the strength of specific markers of depression from a set of 28 speech features.

RESULTS

Increased depressive symptoms were associated with speech rate, articulation rate and intensity of speech elicited from a scripted task. These features had consistently stronger effect sizes than pauses.

LIMITATIONS

Our findings are derived at the cohort level so may have limited impact on identifying intra-individual speech changes associated with changes in symptom severity. The analysis of features averaged over the entire recording may have underestimated the importance of some features.

CONCLUSIONS

Participants with more severe depressive symptoms spoke more slowly and quietly. Our findings are from a real-world, multilingual, clinical dataset so represent a step-change in the usefulness of speech as a digital phenotype of MDD.

Guess What We Can Hear-Novel Voice Biomarkers for the Remote Detection of Disease

The advancement of digital biomarkers and the provision of remote health care greatly progressed during the coronavirus disease 2019 global pandemic. Combining voice/speech data with artificial intelligence and machine-based learning offers a novel solution to the growing demand for telemedicine. Voice biomarkers, obtained from the extraction of characteristic acoustic and linguistic features, are associated with a variety of diseases and even coronavirus disease 2019. In the current review, we (1) describe the basis on which digital voice biomarkers could facilitate "telemedicine," (2) discuss potential mechanisms that may explain the association between voice biomarkers and disease, (3) offer a novel classification system to conceptualize voice biomarkers depending on different methods for recording and analyzing voice/speech samples, (4) outline evidence revealing an association between voice biomarkers and a number of disease states, and (5) describe the process of developing a voice biomarker from recording, storing voice samples, and extracting acoustic and linguistic features relevant to training and testing deep and machine-based learning algorithms to detect disease. We further explore several important future considerations in this area of research, including the necessity for clinical trials and the importance of safeguarding data and individual privacy. To this end, we searched PubMed and Google Scholar to identify studies evaluating the relationship between voice/speech features and biomarkers and various diseases. Search terms included digital biomarker, telemedicine, voice features, voice biomarker, speech features, speech biomarkers, acoustics, linguistics, cardiovascular disease, neurologic disease, psychiatric disease, and infectious disease. The search was limited to studies published in English in peer-reviewed journals between 1980 and the present. To identify potential studies not captured by our database search strategy, we also searched studies listed in the bibliography of relevant publications and reviews.

Depressive symptoms are specifically related to speech pauses in schizophrenia spectrum disorders

Depression is a common and debilitating mental illness associated with sadness and negativity and is often comorbid with other psychiatric conditions, such as schizophrenia. Depressive symptoms are presently primarily assessed through clinical interviews, however there are other behavioural indicators being investigated as more objective methods of depressive symptom assessment. The present study aimed to evaluate the utility of assessing depression using quantitative speech parameters by comparing speech between 23 schizophrenia/schizoaffective patients with clinically significant depressive symptoms (DP) 19 schizophrenia/schizoaffective patients without depressive symptoms (NDP) and 22 healthy controls with no psychiatric history (HC). Participant audio recordings were transcribed and analyzed to extract five types of speech variables: utterances, words, speaking rate, formulation errors and pauses. The results indicated that DP patients produced significantly more pauses within utterances, and had more utterances with pauses compared to NDP patients and HCs (p = <.05), who performed similarly to each other. Word, speaking rate and formulation errors variables were not significantly different between the patient groups (p > .05). The findings suggest that depressive symptoms may have a specific relationship to speech pauses, and support the potential future use of speech pause assessments as an alternative and objective depression rating and monitoring tool.

Digital Phenotyping for Differential Diagnosis of Major Depressive Episode: Narrative Review

BACKGROUND

Major depressive episode (MDE) is a common clinical syndrome. It can be found in different pathologies such as major depressive disorder (MDD), bipolar disorder (BD), posttraumatic stress disorder (PTSD), or even occur in the context of psychological trauma. However, only 1 syndrome is described in international classifications (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition [DSM-5]/International Classification of Diseases 11th Revision [ICD-11]), which do not take into account the underlying pathology at the origin of the MDE. Clinical interviews are currently the best source of information to obtain the etiological diagnosis of MDE. Nevertheless, it does not allow an early diagnosis and there are no objective measures of extracted clinical information. To remedy this, the use of digital tools and their correlation with clinical symptomatology could be useful.

OBJECTIVE

We aimed to review the current application of digital tools for MDE diagnosis while highlighting shortcomings for further research. In addition, our work was focused on digital devices easy to use during clinical interview and mental health issues where depression is common.

METHODS

We conducted a narrative review of the use of digital tools during clinical interviews for MDE by searching papers published in PubMed/MEDLINE, Web of Science, and Google Scholar databases since February 2010. The search was conducted from June to September 2021. Potentially relevant papers were then compared against a checklist for relevance and reviewed independently for inclusion, with focus on 4 allocated topics of (1) automated voice analysis, behavior analysis by (2) video and physiological measures, (3) heart rate variability (HRV), and (4) electrodermal activity (EDA). For this purpose, we were interested in 4 frequently found clinical conditions in which MDE can occur: (1) MDD, (2) BD, (3) PTSD, and (4) psychological trauma.

RESULTS

A total of 74 relevant papers on the subject were qualitatively analyzed and the information was synthesized. Thus, a digital phenotype of MDE seems to emerge consisting of modifications in speech features (namely, temporal, prosodic, spectral, source, and formants) and in speech content, modifications in nonverbal behavior (head, hand, body and eyes movement, facial expressivity, and gaze), and a decrease in physiological measurements (HRV and EDA). We not only found similarities but also differences when MDE occurs in MDD, BD, PTSD, or psychological trauma. However, comparative studies were rare in BD or PTSD conditions, which does not allow us to identify clear and distinct digital phenotypes.

CONCLUSIONS

Our search identified markers from several modalities that hold promise for helping with a more objective diagnosis of MDE. To validate their potential, further longitudinal and prospective studies are needed.

Applications of Speech Analysis in Psychiatry

ABSTRACT

The need for objective measurement in psychiatry has stimulated interest in alternative indicators of the presence and severity of illness. Speech may offer a source of information that bridges the subjective and objective in the assessment of mental disorders. We systematically reviewed the literature for articles exploring speech analysis for psychiatric applications. The utility of speech analysis depends on how accurately speech features represent clinical symptoms within and across disorders. We identified four domains of the application of speech analysis in the literature: diagnostic classification, assessment of illness severity, prediction of onset of illness, and prognosis and treatment outcomes. We discuss the findings in each of these domains, with a focus on how types of speech features characterize different aspects of psychopathology. Models that bring together multiple speech features can distinguish speakers with psychiatric disorders from healthy controls with high accuracy. Differentiating between types of mental disorders and symptom dimensions are more complex problems that expose the transdiagnostic nature of speech features. Convergent progress in speech research and computer sciences opens avenues for implementing speech analysis to enhance objectivity of assessment in clinical practice. Application of speech analysis will need to address issues of ethics and equity, including the potential to perpetuate discriminatory bias through models that learn from clinical assessment data. Methods that mitigate bias are available and should play a key role in the implementation of speech analysis.

Evaluating the utility of daily speech assessments for monitoring depression symptoms

Objective

Depression is a common mental health disorder and a major public health concern, significantly interfering with the lives of those affected. The complex clinical presentation of depression complicates symptom assessments. Day-to-day fluctuations of depression symptoms within an individual bring an additional barrier, since infrequent testing may not reveal symptom fluctuation. Digital measures such as speech can facilitate daily objective symptom evaluation. Here, we evaluated the effectiveness of daily speech assessment in characterizing speech fluctuations in the context of depression symptoms, which can be completed remotely, at a low cost and with relatively low administrative resources.

Methods

Community volunteers (N = 16) completed a daily speech assessment, using the Winterlight Speech App, and Patient Health Questionnaire-9 (PHQ-9) for 30 consecutive business days. We calculated 230 acoustic and 290 linguistic features from individual's speech and investigated their relationship to depression symptoms at the intra-individual level through repeated measures analyses.

Results

We observed that depression symptoms were linked to linguistic features, such as less frequent use of dominant and positive words. Greater depression symptomatology was also significantly correlated with acoustic features: reduced variability in speech intensity and increased jitter.

Conclusions

Our findings support the feasibility of using acoustic and linguistic features as a measure of depression symptoms and propose daily speech assessment as a tool for better characterization of symptom fluctuations.

Speech as a Biomarker for Depression

BACKGROUND

Depression is a debilitating disorder that at present lacks a reliable biomarker to aid in diagnosis and early detection. Recent advances in computational analytic approaches have opened up new avenues in developing such a biomarker by taking advantage of the wealth of information that can be extracted from a person's speech.

OBJECTIVE

The current review provides an overview of the latest findings in the rapidly evolving field of computational language analysis for the detection of depression. We cover a wide range of both acoustic and content-related linguistic features, data types (i.e., spoken and written language), and data sources (i.e., lab settings, social media, and smartphone-based). We put special focus on the current methodological advances with regard to feature extraction and computational modeling techniques. Furthermore, we pay attention to potential hurdles in the implementation of automatic speech analysis.

CONCLUSION

Depressive speech is characterized by several anomalies, such as lower speech rate, less pitch variability and more self-referential speech. With current computational modeling techniques, such features can be used to detect depression with an accuracy of up to 91%. The performance of the models is optimized when machine learning techniques are implemented that suit the type and amount of data. Recent studies now work towards further optimization and generalizability of the computational language models to detect depression. Finally, privacy and ethical issues are of paramount importance to be addressed when automatic speech analysis techniques are further implemented in, for example, smartphones. Altogether, computational speech analysis is well underway towards becoming an effective diagnostic aid for depression.

Detecting subtle signs of depression with automated speech analysis in a non-clinical sample

BACKGROUND

Automated speech analysis has gained increasing attention to help diagnosing depression. Most previous studies, however, focused on comparing speech in patients with major depressive disorder to that in healthy volunteers. An alternative may be to associate speech with depressive symptoms in a non-clinical sample as this may help to find early and sensitive markers in those at risk of depression.

METHODS

We included n = 118 healthy young adults (mean age: 23.5 ± 3.7 years; 77% women) and asked them to talk about a positive and a negative event in their life. Then, we assessed the level of depressive symptoms with a self-report questionnaire, with scores ranging from 0-60. We transcribed speech data and extracted acoustic as well as linguistic features. Then, we tested whether individuals below or above the cut-off of clinically relevant depressive symptoms differed in speech features. Next, we predicted whether someone would be below or above that cut-off as well as the individual scores on the depression questionnaire. Since depression is associated with cognitive slowing or attentional deficits, we finally correlated depression scores with performance in the Trail Making Test.

RESULTS

In our sample, n = 93 individuals scored below and n = 25 scored above cut-off for clinically relevant depressive symptoms. Most speech features did not differ significantly between both groups, but individuals above cut-off spoke more than those below that cut-off in the positive and the negative story. In addition, higher depression scores in that group were associated with slower completion time of the Trail Making Test. We were able to predict with 93% accuracy who would be below or above cut-off. In addition, we were able to predict the individual depression scores with low mean absolute error (3.90), with best performance achieved by a support vector machine.

CONCLUSIONS

Our results indicate that even in a sample without a clinical diagnosis of depression, changes in speech relate to higher depression scores. This should be investigated in more detail in the future. In a longitudinal study, it may be tested whether speech features found in our study represent early and sensitive markers for subsequent depression in individuals at risk.

Identifying psychiatric manifestations in schizophrenia and depression from audio-visual behavioural indicators through a machine-learning approach

Schizophrenia (SCZ) and depression (MDD) are two chronic mental disorders that seriously affect the quality of life of millions of people worldwide. We aim to develop machine-learning methods with objective linguistic, speech, facial, and motor behavioral cues to reliably predict the severity of psychopathology or cognitive function, and distinguish diagnosis groups. We collected and analyzed the speech, facial expressions, and body movement recordings of 228 participants (103 SCZ, 50 MDD, and 75 healthy controls) from two separate studies. We created an ensemble machine-learning pipeline and achieved a balanced accuracy of 75.3% for classifying the total score of negative symptoms, 75.6% for the composite score of cognitive deficits, and 73.6% for the total score of general psychiatric symptoms in the mixed sample containing all three diagnostic groups. The proposed system is also able to differentiate between MDD and SCZ with a balanced accuracy of 84.7% and differentiate patients with SCZ or MDD from healthy controls with a balanced accuracy of 82.3%. These results suggest that machine-learning models leveraging audio-visual characteristics can help diagnose, assess, and monitor patients with schizophrenia and depression.

Establishment and psychometric characteristics of emotional words list for suicidal risk assessment in speech emotion recognition

BACKGROUND

Emotional disturbance is an important risk factor of suicidal behaviors. To ensure speech emotion recognition (SER), a novel technique to evaluate emotional characteristics of speech, precision in labeling emotional words is a prerequisite. Currently, a list of suicide-related emotional word is absent. The aims of this study were to establish an Emotional Words List for Suicidal Risk Assessment (EWLSRA) and test the reliability and validity of the list in a suicide-related SER task.

METHODS

Suicide-related emotion words were nominated and discussed by 10 suicide prevention professionals. Sixty-five tape-recordings of calls to a large psychological support hotline in China were selected to test psychometric characteristics of the EWLSRA.

RESULTS

The results shows that the EWLSRA consists of 11 emotion words which were highly associated with suicide risk scores and suicide attempts. Results of exploratory factor analysis support one-factor model of this list. The Fleiss' Kappa value of 0.42 indicated good inter-rater reliability of the list. In terms of criteria validities, indices of despair (Spearman ρ = 0.54, P < 0.001), sadness (ρ = 0.37, P = 0.006), helplessness (ρ = 0.45, P = 0.001), and numbness (ρ = 0.35, P = 0.009) were significantly associated with suicidal risk scores. The index of the emotional word of numbness in callers with suicide attempt during the 12-month follow-up was significantly higher than that in callers without suicide attempt during the follow-up (P = 0.049).

CONCLUSION

This study demonstrated that the EWLSRA has adequate psychometric performance in identifying suicide-related emotional words of recording of hotline callers to a national wide suicide prevention line. This list can be useful for SER in future studies on suicide prevention.

Depression Severity Detection Using Read Speech with a Divide-and-Conquer Approach

We propose a divide-and-conquer approach to detect depression severity using speech. We divide speech features based on their attributes, i.e., acoustic, prosodic, and language features, then fuse them in a modeling stage with fully connected deep neural networks. Experiments with 76 clinically depressed patients (38 severe and 38 moderate in terms of Montgomery-Asberg Depression Rating Scale (MADRS)), we obtain 78% accuracy while patients' self-reporting scores can classify their own status with 79% accuracy.

Unobtrusive Sensing Technology for Quantifying Stress and Well-Being Using Pulse, Speech, Body Motion, and Electrodermal Data in a Workplace Setting: Study Concept and Design

Introduction: Mental disorders are a leading cause of disability worldwide. Depression has a significant impact in the field of occupational health because it is particularly prevalent during working age. On the other hand, there are a growing number of studies on the relationship between "well-being" and employee productivity. To promote healthy and productive workplaces, this study aims to develop a technique to quantify stress and well-being in a way that does not disturb the workplace. Methods and analysis: This is a single-arm prospective observational study. The target population is adult (>20 years old) workers at companies that often engage in desk work; specifically, a person who sits in front of a computer for at least half their work hours. The following data will be collected: (a) participants' background characteristics; (b) participants' biological data during the 4-week observation period using sensing devices such as a camera built into the computer (pulse wave data extracted from the facial video images), a microphone built into their work computer (voice data), and a wristband-type wearable device (electrodermal activity data, body motion data, and body temperature); (c) stress, well-being, and depression rating scale assessment data. The analysis workflow is as follows: (1) primary analysis, comprised of using software to digitalize participants' vital information; (2) secondary analysis, comprised of examining the relationship between the quantified vital data from (1), stress, well-being, and depression; (3) tertiary analysis, comprised of generating machine learning algorithms to estimate stress, well-being, and degree of depression in relation to each set of vital data as well as multimodal vital data. Discussion: This study will evaluate digital phenotype regarding stress and well-being of white-collar workers over a 4-week period using persistently obtainable biomarkers such as heart rate, acoustic characteristics, body motion, and electrodermal activity. Eventually, this study will lead to the development of a machine learning algorithm to determine people's optimal levels of stress and well-being. Ethics and dissemination: Collected data and study results will be disseminated widely through conference presentations, journal publications, and/or mass media. The summarized results of our overall analysis will be supplied to participants. Registration: UMIN000036814.