User Feedback on the Use of a Natural Language Processing Application to Screen for Suicide Risk in the Emergency Department

Suicide is the 10th leading cause of death in the USA and globally. Despite decades of research, the ability to predict who will die by suicide is still no better than 50%. Traditional screening instruments have helped identify risk factors for suicide, but they have not provided accurate predictive power for reducing death rates. Over the past decade, natural language processing (NLP), a form of machine learning (ML), has been used to identify suicide risk by analyzing language data. Recent work has demonstrated the successful integration of a suicide risk screening interview to collect language data for NLP analysis from patients in two emergency departments (ED) of a large healthcare system. Results indicated that ML/NLP models performed well identifying patients that came to the ED for suicide risk. However, little is known about the clinician's perspective of how a qualitative brief interview suicide risk screening tool to collect language data for NLP integrates into an ED workflow. This report highlights the feedback and observations of patient experiences obtained from clinicians using brief suicide screening interviews. The investigator used an open-ended, narrative interview approach to inquire about the qualitative interview process. Three overarching themes were identified: behavioral health workflow, clinical implications of interview probes, and integration of an application into provider patient experience. Results suggest a brief, qualitative interview method was feasible, person-centered, and useful as a suicide risk detection approach.

A multimodal dialog approach to mental state characterization in clinically depressed, anxious, and suicidal populations

Background

The rise of depression, anxiety, and suicide rates has led to increased demand for telemedicine-based mental health screening and remote patient monitoring (RPM) solutions to alleviate the burden on, and enhance the efficiency of, mental health practitioners. Multimodal dialog systems (MDS) that conduct on-demand, structured interviews offer a scalable and cost-effective solution to address this need.

Objective

This study evaluates the feasibility of a cloud based MDS agent, Tina, for mental state characterization in participants with depression, anxiety, and suicide risk.

Method

Sixty-eight participants were recruited through an online health registry and completed 73 sessions, with 15 (20.6%), 21 (28.8%), and 26 (35.6%) sessions screening positive for depression, anxiety, and suicide risk, respectively using conventional screening instruments. Participants then interacted with Tina as they completed a structured interview designed to elicit calibrated, open-ended responses regarding the participants' feelings and emotional state. Simultaneously, the platform streamed their speech and video recordings in real-time to a HIPAA-compliant cloud server, to compute speech, language, and facial movement-based biomarkers. After their sessions, participants completed user experience surveys. Machine learning models were developed using extracted features and evaluated with the area under the receiver operating characteristic curve (AUC).

Results

For both depression and suicide risk, affected individuals tended to have a higher percent pause time, while those positive for anxiety showed reduced lip movement relative to healthy controls. In terms of single-modality classification models, speech features performed best for depression (AUC = 0.64; 95% CI = 0.51-0.78), facial features for anxiety (AUC = 0.57; 95% CI = 0.43-0.71), and text features for suicide risk (AUC = 0.65; 95% CI = 0.52-0.78). Best overall performance was achieved by decision fusion of all models in identifying suicide risk (AUC = 0.76; 95% CI = 0.65-0.87). Participants reported the experience comfortable and shared their feelings.

Conclusion

MDS is a feasible, useful, effective, and interpretable solution for RPM in real-world clinical depression, anxiety, and suicidal populations. Facial information is more informative for anxiety classification, while speech and language are more discriminative of depression and suicidality markers. In general, combining speech, language, and facial information improved model performance on all classification tasks.

Virtually screening adults for depression, anxiety, and suicide risk using machine learning and language from an open-ended interview

Background

Current depression, anxiety, and suicide screening techniques rely on retrospective patient reported symptoms to standardized scales. A qualitative approach to screening combined with the innovation of natural language processing (NLP) and machine learning (ML) methods have shown promise to enhance person-centeredness while detecting depression, anxiety, and suicide risk from in-the-moment patient language derived from an open-ended brief interview.

Objective

To evaluate the performance of NLP/ML models to identify depression, anxiety, and suicide risk from a single 5-10-min semi-structured interview with a large, national sample.

Method

Two thousand four hundred sixteen interviews were conducted with 1,433 participants over a teleconference platform, with 861 (35.6%), 863 (35.7%), and 838 (34.7%) sessions screening positive for depression, anxiety, and suicide risk, respectively. Participants completed an interview over a teleconference platform to collect language about the participants' feelings and emotional state. Logistic regression (LR), support vector machine (SVM), and extreme gradient boosting (XGB) models were trained for each condition using term frequency-inverse document frequency features from the participants' language. Models were primarily evaluated with the area under the receiver operating characteristic curve (AUC).

Results

The best discriminative ability was found when identifying depression with an SVM model (AUC = 0.77; 95% CI = 0.75-0.79), followed by anxiety with an LR model (AUC = 0.74; 95% CI = 0.72-0.76), and an SVM for suicide risk (AUC = 0.70; 95% CI = 0.68-0.72). Model performance was generally best with more severe depression, anxiety, or suicide risk. Performance improved when individuals with lifetime but no suicide risk in the past 3 months were considered controls.

Conclusion

It is feasible to use a virtual platform to simultaneously screen for depression, anxiety, and suicide risk using a 5-to-10-min interview. The NLP/ML models performed with good discrimination in the identification of depression, anxiety, and suicide risk. Although the utility of suicide risk classification in clinical settings is still undetermined and suicide risk classification had the lowest performance, the result taken together with the qualitative responses from the interview can better inform clinical decision-making by providing additional drivers associated with suicide risk.

Integration and Validation of a Natural Language Processing Machine Learning Suicide Risk Prediction Model Based on Open-Ended Interview Language in the Emergency Department

BACKGROUND

Emergency departments (ED) are an important intercept point for identifying suicide risk and connecting patients to care, however, more innovative, person-centered screening tools are needed. Natural language processing (NLP) -based machine learning (ML) techniques have shown promise to assess suicide risk, although whether NLP models perform well in differing geographic regions, at different time periods, or after large-scale events such as the COVID-19 pandemic is unknown.

OBJECTIVE

To evaluate the performance of an NLP/ML suicide risk prediction model on newly collected language from the Southeastern United States using models previously tested on language collected in the Midwestern US.

METHOD

37 Suicidal and 33 non-suicidal patients from two EDs were interviewed to test a previously developed suicide risk prediction NLP/ML model. Model performance was evaluated with the area under the receiver operating characteristic curve (AUC) and Brier scores.

RESULTS

NLP/ML models performed with an AUC of 0.81 (95% CI: 0.71-0.91) and Brier score of 0.23.

CONCLUSION

The language-based suicide risk model performed with good discrimination when identifying the language of suicidal patients from a different part of the US and at a later time period than when the model was originally developed and trained.

Interprofessional Suicide Prevention Education

The COVID-19 pandemic has raised concerns regarding possible spikes in suicidal behavior in light of heightened risk factors such as social isolation and financial strain; thus, comprehensive suicide prevention training for emerging health service providers is increasingly vital. This article summarizes an interprofessional education (IPE) suicide prevention course delivered in-person in Spring 2020. Pilot data demonstrate that despite the impact of COVID-19 on higher education, this course had long-term impacts on trainee suicide prevention efficacy, IPE attitudes, and use of course content in practice. Discussion serves to address enhancements for interprofessional and suicide prevention education during and after the pandemic. Emphasis is placed on adaptable training strategies, considerations in the delivery format, guidelines for intensive virtual meetings with trainee teams, and future directions in IPE suicide prevention training research.

A Feasibility Study Using a Machine Learning Suicide Risk Prediction Model Based on Open-Ended Interview Language in Adolescent Therapy Sessions

Background: As adolescent suicide rates continue to rise, innovation in risk identification is warranted. Machine learning can identify suicidal individuals based on their language samples. This feasibility pilot was conducted to explore this technology’s use in adolescent therapy sessions and assess machine learning model performance. Method: Natural language processing machine learning models to identify level of suicide risk using a smartphone app were tested in outpatient therapy sessions. Data collection included language samples, depression and suicidality standardized scale scores, and therapist impression of the client’s mental state. Previously developed models were used to predict suicidal risk. Results: 267 interviews were collected from 60 students in eight schools by ten therapists, with 29 students indicating suicide or self-harm risk. During external validation, models were trained on suicidal speech samples collected from two separate studies. We found that support vector machines (AUC: 0.75; 95% CI: 0.69–0.81) and logistic regression (AUC: 0.76; 95% CI: 0.70–0.82) lead to good discriminative ability, with an extreme gradient boosting model performing the best (AUC: 0.78; 95% CI: 0.72–0.84). Conclusion: Voice collection technology and associated procedures can be integrated into mental health therapists’ workflow. Collected language samples could be classified with good discrimination using machine learning methods.

Integrating Interprofessional Education into Suicide Prevention Training: Results from A Pilot Evaluation

Suicide prevention training for health professions students is lacking, often occurring in disciplinary silos. The present study reports outcomes from an interprofessional education (IPE)-based suicide prevention course for health professions students across a variety of disciplines (e.g., social work, counseling, public health). Using a quasi-experimental design, students either took part in a fully online or blended version of the course. Primary outcomes included: (1) significant moderate-to-large positive gains in suicide prevention knowledge, perceived clinical care skills, and perceived ability to help self-harming patients; (2) moderate positive shifts in sensitivity to risk factors of those who died by suicide; (3) non-significant impacts on IPE-related outcomes; (4) overall high course satisfaction; and (5) students in the blended course preferred several interactive methods more than students in the online course version (large effects). Recommendations are provided for course revision and future implementation in educational and community-based settings.

Managing physical and mental health conditions: Consumer perspectives on integrated care

BACKGROUND

Despite the growing trend of integrating primary care and mental health services, little research has documented how consumers with severe mental illnesses manage comorbid conditions or view integrated services.

OBJECTIVES

We sought to better understand how consumers perceive and manage both mental and physical health conditions and their views of integrated services.

METHODS

We conducted semi-structured interviews with consumers receiving primary care services integrated in a community mental health setting.

RESULTS

Consumers described a range of strategies to deal with physical health conditions and generally viewed mental and physical health conditions as impacting one another. Consumers viewed integration of primary care and mental health services favorably, specifically its convenience, friendliness and knowledge of providers, and collaboration between providers.

CONCLUSIONS

Although integration was viewed positively, consumers with SMI may need a myriad of strategies and supports to both initiate and sustain lifestyle changes that address common physical health problems.

A comparison of self-reported and phone-administered methods of ACT fidelity assessment: a pilot study in Indiana

OBJECTIVE

Monitoring fidelity of assertive community treatment (ACT) teams is costly. This study investigated the reliability and validity of a less burdensome approach: self-reported assessment.

METHODS

Phone-administered and self-reported assessments were compared for 16 ACT teams. Team leaders completed a self-report protocol providing information sufficient to score the Dartmouth Assertive Community Treatment Scale (DACTS). Two raters scored the DACTS using only self-reported information. Two additional raters conducted phone interviews with team leaders, verifying the self-reported data, and independently scored the DACTS.

RESULTS

DACTS total scores obtained via self-reported assessments were reliable and valid compared with phone-administered assessment on the basis of interrater consistency (intraclass correlation) and consensus (mean rating differences). Phone-administered assessments agreed with self-reported assessments within .25 scale points (out of 5 points) for 15 of 16 teams.

CONCLUSIONS

A self-report approach could address concerns regarding costs of monitoring as part of a stepped approach to quality assurance.