Remote Digital Measurement of Facial and Vocal Markers of Major Depressive Disorder Severity and Treatment Response: A Pilot Study

Day-to-day dynamics of facial emotion expressions in posttraumatic stress disorder

Facial expressions are an essential component of emotions that may reveal mechanisms maintaining posttraumatic stress disorder (PTSD). However, most research on emotions in PTSD has relied on self-reports, which only capture subjective affect. The few studies on outward emotion expressions have been hampered by methodological limitations, including low ecological validity and failure to capture the dynamic nature of emotions and symptoms. Our study addresses these limitations with an approach that has not been applied to psychopathology: person-specific models of day-to-day facial emotion expression and PTSD symptom dynamics. We studied a sample of World Trade Center responders (N = 112) with elevated PTSD pathology who recorded a daily video diary and self-reported symptoms for 90 days (8953 videos altogether). Facial expressions were detected from video recordings with a facial emotion recognition model. In data-driven, idiographic network models, most participants (80 %) had at least one, reliable expression-symptom link. Six expression-symptom dynamics were significant for >10 % of the sample. Each of these dynamics had statistically meaningful heterogeneity, with some people's symptoms related to over-expressivity and others to under-expressivity. Our results provide the foundation for a more complete understanding of emotions in PTSD that not only includes subjective feelings but also outward emotion expressions.

Initial expressed emotion during neuropsychological assessment: investigating motivational dimensions of approach and avoidance

OBJECTIVE

Prior work indicates that discrete emotions are linked to performance across multiple domains of cognitive function and thus have the potential to impact cognitive profiles in neuropsychological assessment. However, reported presence and magnitude of the relationships between emotion and cognitive test performance are inconsistent. Variable findings in this regard could be due to failure to consider motivations associated with expressed emotion. To better understand the potential impact of expressed emotion on neuropsychological test performance, it may be beneficial to consider approach and avoidance motivation during assessment.

METHOD

The current cross-sectional study examined associations between cognitive performance and digitally phenotyped facial expressions of discrete emotions on dimensions of approach (i.e. joy, sadness, anger) and avoidance (i.e. fear, disgust) in the context of virtual neuropsychological assessment in 104 adults (ages 55-90).

RESULTS

Initial facial expressions categorized as anger and joy predicted later reduced cognitive performance in aspects of memory and executive function within the virtual session, respectively. Test performance was associated neither with sadness nor with avoidance emotions (i.e. disgust or fear).

CONCLUSIONS

Results of the current study did not strongly align with approach/avoidance explanations for links between emotion and cognitive performance; however, results might support an arousal-based explanation, as joy and anger are both high arousal emotions. Additional investigation is needed to understand the intersection of emotion motivation and physiological arousal in the context of neuropsychological assessment.

Beyond Detection: Towards Actionable Sensing Research in Clinical Mental Healthcare

Researchers in ubiquitous computing have long promised that passive sensing will revolutionize mental health measurement by detecting individuals in a population experiencing a mental health disorder or specific symptoms. Recent work suggests that detection tools do not generalize well when trained and tested in more heterogeneous samples. In this work, we contribute a narrative review and findings from two studies with 41 mental health clinicians to understand these generalization challenges. Our findings motivate research on actionable sensing, as an alternative to detection research, studying how passive sensing can augment traditional mental health measures to support actions in clinical care. Specifically, we identify how passive sensing can support clinical actions by revealing patients' presenting problems for treatment and identifying targets for behavior change and symptom reduction, but passive data requires additional contextual information to be appropriately interpreted and used in care. We conclude by suggesting research at the intersection of actionable sensing and mental healthcare, to align technical research in ubiquitous computing with clinical actions and needs.

Digital assessment of nonverbal behaviors forecasts first onset of depression

BACKGROUND

Adolescence is marked by a sharp increase in the incidence of depression, especially in females. Identification of risk for depressive disorders (DD) in this key developmental stage can help prevention efforts, mitigating the clinical and public burden of DD. While frequently used in diagnosis, nonverbal behaviors are relatively understudied as risk markers for DD. Digital technology, such as facial recognition, may provide objective, fast, efficient, and cost-effective means of measuring nonverbal behavior.

METHOD

Here, we analyzed video-recorded clinical interviews of 359 never-depressed adolescents females via commercially available facial emotion recognition software.

RESULTS

We found that average head and facial movements forecast future first onset of depression (AUC = 0.70) beyond the effects of other established self-report and physiological markers of DD risk.

CONCLUSIONS

Overall, these findings suggest that digital assessment of nonverbal behaviors may provide a promising risk marker for DD, which could aid in early identification and intervention efforts.

Behavioral meaures of psychotic disorders: Using automatic facial coding to detect nonverbal expressions in video

Emotional deficits in psychosis are prevalent and difficult to treat. In particular, much remains unknown about facial expression abnormalities, and a key reason is that expressions are very labor-intensive to code. Automatic facial coding (AFC) can remove this barrier. The current study sought to both provide evidence for the utility of AFC in psychosis for research purposes and to provide evidence that AFC are valid measures of clinical constructs. Changes of facial expressions and head position of participants-39 with schizophrenia/schizoaffective disorder (SZ), 46 with other psychotic disorders (OP), and 108 never psychotic individuals (NP)-were assessed via FaceReader, a commercially available automated facial expression analysis software, using video recorded during a clinical interview. We first examined the behavioral measures of the psychotic disorder groups and tested if they can discriminate between the groups. Next, we evaluated links of behavioral measures with clinical symptoms, controlling for group membership. We found the SZ group was characterized by significantly less variation in neutral expressions, happy expressions, arousal, and head movements compared to NP. These measures discriminated SZ from NP well (AUC = 0.79, sensitivity = 0.79, specificity = 0.67) but discriminated SZ from OP less well (AUC = 0.66, sensitivity = 0.77, specificity = 0.46). We also found significant correlations between clinician-rated symptoms and most behavioral measures (particularly happy expressions, arousal, and head movements). Taken together, these results suggest that AFC can provide useful behavioral measures of psychosis, which could improve research on non-verbal expressions in psychosis and, ultimately, enhance treatment.

Associations between keystroke and stylus metadata and depressive symptoms in adolescents

BACKGROUND

Adolescents often experience a heightened incidence of depressive symptoms, which can persist without early intervention. However, adolescents often struggle to identify depressive symptoms, and even when they are aware of these symptoms, seeking help is not always their immediate response. This study aimed to explore the relationship between passively collected digital data, specifically keystroke and stylus data collected via mobile devices, and the manifestation of depressive symptoms.

METHODS

A total of 927 first-year middle school students from schools in Seoul solved Korean language and math problems. Throughout this study, 77 types of keystroke and stylus data were collected, including parameters such as the number of key presses, tap pressure, stroke speed, and stroke acceleration. Depressive symptoms were measured using the self-rated Patient Health Questionnaire-9 (PHQ-9).

RESULTS

Multiple regression analysis highlighted the significance of stroke length, speed, and acceleration, the average y-coordinate, the tap pressure, and the number of incorrect answers in relation to PHQ-9 scores. The keystroke and stylus metadata were able to reflect mood, energy, cognitive abilities, and psychomotor symptoms among adolescents with depressive symptoms.

CONCLUSIONS

This study demonstrates the potential of automatically collected data during school exams or classes for the early screening of clinical depressive symptoms in students. This study has the potential to serve as a cornerstone in the development of digital data frameworks for the early detection of depressive symptoms in adolescents.

From smartphone data to clinically relevant predictions: A systematic review of digital phenotyping methods in depression

BACKGROUND

Smartphone-based digital phenotyping enables potentially clinically relevant information to be collected as individuals go about their day. This could improve monitoring and interventions for people with Major Depressive Disorder (MDD). The aim of this systematic review was to investigate current digital phenotyping features and methods used in MDD.

METHODS

We searched PubMed, PsycINFO, Embase, Scopus and Web of Science (10/11/2023) for articles including: (1) MDD population, (2) smartphone-based features, (3) validated ratings. Risk of bias was assessed using several sources. Studies were compared within analysis goals (correlating features with depression, predicting symptom severity, diagnosis, mood state/episode, other). Twenty-four studies (9801 participants) were included.

RESULTS

Studies achieved moderate performance. Common themes included challenges from complex and missing data (leading to a risk of bias), and a lack of external validation.

DISCUSSION

Studies made progress towards relating digital phenotypes to clinical variables, often focusing on time-averaged features. Methods investigating temporal dynamics more directly may be beneficial for patient monitoring. European Research Council consolidator grant: 101001118, Prospero: CRD42022346264, Open Science Framework: https://osf.io/s7ay4.

Use of voice features from smartphones for monitoring depressive disorders: Scoping review

Object

This review evaluates the use of smartphone-based voice data for predicting and monitoring depression.

Methods

A scoping review was conducted, examining 14 studies from Medline, Scopus, and Web of Science (2010-2023) on voice data collection methods and the use of voice features for minitoring depression.

Results

Voice data, especially prosodic features like fundamental frequency and pitch, show promise for predicting depression, though their sole predictive power requires further validation. Integrating voice with multimodal sensor data has been shown to improve accuracy significantly.

Conclusion

Smartphone-based voice monitoring offers a promising, noninvasive, and cost-effective approach to depression management. The integration of machine learning with sensor data could significantly enhance mental health monitoring, necessitating further research and longitudinal studies for validation.

Digital Phenotyping for Monitoring Mental Disorders: Systematic Review

BACKGROUND

The COVID-19 pandemic has increased the impact and spread of mental illness and made health services difficult to access; therefore, there is a need for remote, pervasive forms of mental health monitoring. Digital phenotyping is a new approach that uses measures extracted from spontaneous interactions with smartphones (eg, screen touches or movements) or other digital devices as markers of mental status.

OBJECTIVE

This review aimed to evaluate the feasibility of using digital phenotyping for predicting relapse or exacerbation of symptoms in patients with mental disorders through a systematic review of the scientific literature.

METHODS

Our research was carried out using 2 bibliographic databases (PubMed and Scopus) by searching articles published up to January 2023. By following the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analysis) guidelines, we started from an initial pool of 1150 scientific papers and screened and extracted a final sample of 29 papers, including studies concerning clinical populations in the field of mental health, which were aimed at predicting relapse or exacerbation of symptoms. The systematic review has been registered on the web registry Open Science Framework.

RESULTS

We divided the results into 4 groups according to mental disorder: schizophrenia (9/29, 31%), mood disorders (15/29, 52%), anxiety disorders (4/29, 14%), and substance use disorder (1/29, 3%). The results for the first 3 groups showed that several features (ie, mobility, location, phone use, call log, heart rate, sleep, head movements, facial and vocal characteristics, sociability, social rhythms, conversations, number of steps, screen on or screen off status, SMS text message logs, peripheral skin temperature, electrodermal activity, light exposure, and physical activity), extracted from data collected via the smartphone and wearable wristbands, can be used to create digital phenotypes that could support gold-standard assessment and could be used to predict relapse or symptom exacerbations.

CONCLUSIONS

Thus, as the data were consistent for almost all the mental disorders considered (mood disorders, anxiety disorders, and schizophrenia), the feasibility of this approach was confirmed. In the future, a new model of health care management using digital devices should be integrated with the digital phenotyping approach and tailored mobile interventions (managing crises during relapse or exacerbation).

Multilingual markers of depression in remotely collected speech samples: A preliminary analysis

BACKGROUND

Speech contains neuromuscular, physiological and cognitive components, and so is a potential biomarker of mental disorders. Previous studies indicate that speaking rate and pausing are associated with major depressive disorder (MDD). However, results are inconclusive as many studies are small and underpowered and do not include clinical samples. These studies have also been unilingual and use speech collected in controlled settings. If speech markers are to help understand the onset and progress of MDD, we need to uncover markers that are robust to language and establish the strength of associations in real-world data.

METHODS

We collected speech data in 585 participants with a history of MDD in the United Kingdom, Spain, and Netherlands as part of the RADAR-MDD study. Participants recorded their speech via smartphones every two weeks for 18 months. Linear mixed models were used to estimate the strength of specific markers of depression from a set of 28 speech features.

RESULTS

Increased depressive symptoms were associated with speech rate, articulation rate and intensity of speech elicited from a scripted task. These features had consistently stronger effect sizes than pauses.

LIMITATIONS

Our findings are derived at the cohort level so may have limited impact on identifying intra-individual speech changes associated with changes in symptom severity. The analysis of features averaged over the entire recording may have underestimated the importance of some features.

CONCLUSIONS

Participants with more severe depressive symptoms spoke more slowly and quietly. Our findings are from a real-world, multilingual, clinical dataset so represent a step-change in the usefulness of speech as a digital phenotype of MDD.

Effectiveness of a scalable, remotely delivered stepped-care intervention to reduce symptoms of psychological distress among Polish migrant workers in the Netherlands: study protocol for the RESPOND randomised controlled trial

BACKGROUND

The COVID-19 pandemic has negatively affected the mental health of international migrant workers (IMWs). IMWs experience multiple barriers to accessing mental health care. Two scalable interventions developed by the World Health Organization (WHO) were adapted to address some of these barriers: Doing What Matters in times of stress (DWM), a guided self-help web application, and Problem Management Plus (PM +), a brief facilitator-led program to enhance coping skills. This study examines whether DWM and PM + remotely delivered as a stepped-care programme (DWM/PM +) is effective and cost-effective in reducing psychological distress, among Polish migrant workers with psychological distress living in the Netherlands.

METHODS

The stepped-care DWM/PM + intervention will be tested in a two-arm, parallel-group, randomized controlled trial (RCT) among adult Polish migrant workers with self-reported psychological distress (Kessler Psychological Distress Scale; K10 > 15.9). Participants (n = 212) will be randomized into either the intervention group that receives DWM/PM + with psychological first aid (PFA) and care-as-usual (enhanced care-as-usual or eCAU), or into the control group that receives PFA and eCAU-only (1:1 allocation ratio). Baseline, 1-week post-DWM (week 7), 1-week post-PM + (week 13), and follow-up (week 21) self-reported assessments will be conducted. The primary outcome is psychological distress, assessed with the Patient Health Questionnaire Anxiety and Depression Scale (PHQ-ADS). Secondary outcomes are self-reported symptoms of depression, anxiety, posttraumatic stress disorder (PTSD), resilience, quality of life, and cost-effectiveness. In a process evaluation, stakeholders' views on barriers and facilitators to the implementation of DWM/PM + will be evaluated.

DISCUSSION

To our knowledge, this is one of the first RCTs that combines two scalable, psychosocial WHO interventions into a stepped-care programme for migrant populations. If proven to be effective, this may bridge the mental health treatment gap IMWs experience.

TRIAL REGISTRATION

Dutch trial register NL9630, 20/07/2021, https://www.onderzoekmetmensen.nl/en/trial/27052.

Recommendations for selection and adaptation of rating scales for clinical studies of rapid-acting antidepressants

The novel mechanisms of action (MOA) derived from some recently introduced molecular targets have led to regulatory approvals for rapid acting antidepressants (RAADs) that can generate responses within hours or days, rather than weeks or months. These novel targets include the N-methyl-D-glutamate receptor antagonist ketamine, along with its enantiomers and various derivatives, and the allosteric modulators of gamma-aminobutyric acid (GABA) receptors. There has also been a strong resurgence in interest in psychedelic compounds that impact a range of receptor sites including D1, 5-HT7, KOR, 5-HT5A, Sigma-1, NMDA, and BDNF. The RAADs developed from these novel targets have enabled successful treatment for difficult to treat depressed individuals and has generated a new wave of innovation in research and treatment. Despite the advances in the neurobiology and clinical treatment of mood disorders, we are still using rating instruments that were created decades ago for drugs from a different era (e.g., The Hamilton and Montgomery-Åsberg depression rating scales, HDRS, and MADRS) continue to be used. These rating instruments were designed to assess mood symptoms over a 7-day time frame. Consequently, the use of these rating instruments often requires modifications to address items that cannot be assessed in short time frames, such as the sleep and appetite items. This review describes the adaptative approaches that have been made with the existing scales to meet this need and examines additional domains such as daily activities, side effects, suicidal ideation and behavior, and role functioning. Recommendations for future studies are described, including the challenges related to implementation of these adapted measures and approaches to mitigation.

Machine Learning and the Digital Measurement of Psychological Health

Since its inception, the discipline of psychology has utilized empirical epistemology and mathematical methodologies to infer psychological functioning from direct observation. As new challenges and technological opportunities emerge, scientists are once again challenged to define measurement paradigms for psychological health and illness that solve novel problems and capitalize on new technological opportunities. In this review, we discuss the theoretical foundations of and scientific advances in remote sensor technology and machine learning models as they are applied to quantify psychological functioning, draw clinical inferences, and chart new directions in treatment.

Evaluating the utility of daily speech assessments for monitoring depression symptoms

Objective

Depression is a common mental health disorder and a major public health concern, significantly interfering with the lives of those affected. The complex clinical presentation of depression complicates symptom assessments. Day-to-day fluctuations of depression symptoms within an individual bring an additional barrier, since infrequent testing may not reveal symptom fluctuation. Digital measures such as speech can facilitate daily objective symptom evaluation. Here, we evaluated the effectiveness of daily speech assessment in characterizing speech fluctuations in the context of depression symptoms, which can be completed remotely, at a low cost and with relatively low administrative resources.

Methods

Community volunteers (N = 16) completed a daily speech assessment, using the Winterlight Speech App, and Patient Health Questionnaire-9 (PHQ-9) for 30 consecutive business days. We calculated 230 acoustic and 290 linguistic features from individual's speech and investigated their relationship to depression symptoms at the intra-individual level through repeated measures analyses.

Results

We observed that depression symptoms were linked to linguistic features, such as less frequent use of dominant and positive words. Greater depression symptomatology was also significantly correlated with acoustic features: reduced variability in speech intensity and increased jitter.

Conclusions

Our findings support the feasibility of using acoustic and linguistic features as a measure of depression symptoms and propose daily speech assessment as a tool for better characterization of symptom fluctuations.

Modern views of machine learning for precision psychiatry

In light of the National Institute of Mental Health (NIMH)'s Research Domain Criteria (RDoC), the advent of functional neuroimaging, novel technologies and methods provide new opportunities to develop precise and personalized prognosis and diagnosis of mental disorders. Machine learning (ML) and artificial intelligence (AI) technologies are playing an increasingly critical role in the new era of precision psychiatry. Combining ML/AI with neuromodulation technologies can potentially provide explainable solutions in clinical practice and effective therapeutic treatment. Advanced wearable and mobile technologies also call for the new role of ML/AI for digital phenotyping in mobile mental health. In this review, we provide a comprehensive review of ML methodologies and applications by combining neuroimaging, neuromodulation, and advanced mobile technologies in psychiatry practice. We further review the role of ML in molecular phenotyping and cross-species biomarker identification in precision psychiatry. We also discuss explainable AI (XAI) and neuromodulation in a closed human-in-the-loop manner and highlight the ML potential in multi-media information extraction and multi-modal data fusion. Finally, we discuss conceptual and practical challenges in precision psychiatry and highlight ML opportunities in future research.

Measuring Health-Related Quality of Life With Multimodal Data: Viewpoint

The ability to objectively measure aspects of performance and behavior is a fundamental pillar of digital health, enabling digital wellness products, decentralized trial concepts, evidence generation, digital therapeutics, and more. Emerging multimodal technologies capable of measuring several modalities simultaneously and efforts to integrate inputs across several sources are further expanding the limits of what digital measures can assess. Experts from the field of digital health were convened as part of a multi-stakeholder workshop to examine the progress of multimodal digital measures in two key areas: detection of disease and the measurement of meaningful aspects of health relevant to the quality of life. Here we present a meeting report, summarizing key discussion points, relevant literature, and finally a vision for the immediate future, including how multimodal measures can provide value to stakeholders across drug development and care delivery, as well as three key areas where headway will need to be made if we are to continue to build on the encouraging progress so far: collaboration and data sharing, removal of barriers to data integration, and alignment around robust modular evaluation of new measurement capabilities.