Validating a predictive algorithm for suicide risk with Alaska Native populations

Evaluating the ability of artificial intelligence to predict suicide: A systematic review of reviews

INTRODUCTION

Suicide remains a critical global public health issue, with approximately 800,000 deaths annually. Despite various prevention efforts, suicide rates are rising, highlighting the need for more effective strategies. Traditional suicide risk assessment methods often fall short in accuracy and predictive capability. This has driven interest in artificial intelligence (AI), particularly machine learning (ML), as a potential solution. This paper reviews systematic evaluations of AI's effectiveness in predicting suicide risk, aiming to explore AI's potential while addressing its challenges and limitations.

METHODOLOGY

A meta-research approach was used to review existing systematic reviews on AI's role in suicide risk prediction. Following PRISMA guidelines, a comprehensive search was conducted in PubMed and Web of Science for publications from 2004 to 2024. Relevant studies were selected based on specific inclusion criteria, and data were extracted on review characteristics, AI techniques, outcomes, and methodological quality. The review focuses on AI/ML models predicting suicidal ideation (SI), suicide attempts (SA), and suicide deaths (SD) separately, excluding non-suicidal self-injury.

RESULTS

Out of 96 initial articles, 23 met the inclusion criteria for full-text review. Most studies focused on developing ML models to identify suicide risk, showing promising results in enhancing accuracy and effectiveness. These models utilize various data sources and analytical techniques. However, challenges remain, including high bias risk and issues with interpretability, which necessitate further validation and refinement of AI-driven methods.

CONCLUSION

The review underscores the significant potential of AI, especially ML, in predicting suicide risk and attempts. Although ML models show promise, challenges like data limitations, bias, and interpretability issues need addressing. Continued research and ethical scrutiny are crucial to fully realize AI's potential in suicide prevention.

Accuracy of Self-Report Questionnaires and Records-Based Risk Scores to Identify Adolescents' Risk for Self-Harm

OBJECTIVE

This study aimed to evaluate screening strategies for identifying risk for self-harm among adolescents making outpatient health care visits.

METHODS

Health system records were used to identify a prospective cohort of adolescents completing the Patient Health Questionnaire-9 (PHQ-9) at outpatient visits between October 1, 2015, and March 15, 2020, and a retrospective cohort of adolescents experiencing self-harm events (ascertained from health records and state mortality data) during the same period. Self-harm risk scores were computed from health records. Analyses of the prospective sample examined the sensitivity and positive predictive value (PPV) of questionnaires and risk scores, separately and in combination. Analyses of the retrospective sample examined the proportion of self-harm events that could have been detected by different screening strategies.

RESULTS

The prospective sample (N=8,929) included 43,548 questionnaires, with 1,045 questionnaires followed by a self-harm event within 180 days. A score of ≥2 on PHQ-9 item 9 had a sensitivity of 0.37 and a PPV of 0.09 for self-harm within 180 days of a mental health specialty visit, with similar results for primary care visits. In the retrospective sample, 89% of adolescents made a mental health specialty visit or a primary care visit with a recorded psychiatric diagnosis in the 180 days before a self-harm event.

CONCLUSIONS

Responses to PHQ-9 item 9 and risk scores computed from health records accurately identified adolescents needing additional assessment for risk for self-harm. Over 80% of adolescents experiencing self-harm could have been identified by screening during an outpatient health care visit.

Developing a suicide risk model for use in the Indian Health Service

We developed and evaluated an electronic health record (EHR)-based model for suicide risk specific to an American Indian patient population. Using EHR data for all patients over 18 with a visit between 1/1/2017 and 10/2/2021, we developed a model for the risk of a suicide attempt or death in the 90 days following a visit. Features included demographics, medications, diagnoses, and scores from relevant screening tools. We compared the predictive performance of logistic regression and random forest models against existing suicide screening, which was augmented to include the history of previous attempts or ideation. During the study, 16,835 patients had 331,588 visits, with 490 attempts and 37 deaths by suicide. The logistic regression and random forest models (area under the ROC (AUROC) 0.83 [0.80-0.86]; both models) performed better than enhanced screening (AUROC 0.64 [0.61-0.67]). These results suggest that an EHR-based suicide risk model can add value to existing practices at Indian Health Service clinics.

Performance of Machine Learning Suicide Risk Models in an American Indian Population

Importance

Few suicide risk identification tools have been developed specifically for American Indian and Alaska Native populations, even though these populations face the starkest suicide-related inequities.

Objective

To examine the accuracy of existing machine learning models in a majority American Indian population.

Design, Setting, and Participants

This prognostic study used secondary data analysis of electronic health record data collected from January 1, 2017, to December 31, 2021. Existing models from the Mental Health Research Network (MHRN) and Vanderbilt University (VU) were fitted. Models were compared with an augmented screening indicator that included any previous attempt, recent suicidal ideation, or a recent positive suicide risk screen result. The comparison was based on the area under the receiver operating characteristic curve (AUROC). The study was performed in partnership with a tribe and local Indian Health Service (IHS) in the Southwest. All patients were 18 years or older with at least 1 encounter with the IHS unit during the study period. Data were analyzed between October 6, 2022, and July 29, 2024.

Exposures

Suicide attempts or deaths within 90 days.

Main Outcomes and Measures

Model performance was compared based on the ability to distinguish between those with a suicide attempt or death within 90 days of their last IHS visit with those without this outcome.

Results

Of 16 835 patients (mean [SD] age, 40.0 [17.5] years; 8660 [51.4%] female; 14 251 [84.7%] American Indian), 324 patients (1.9%) had at least 1 suicide attempt, and 37 patients (0.2%) died by suicide. The MHRN model had an AUROC value of 0.81 (95% CI, 0.77-0.85) for 90-day suicide attempts, whereas the VU model had an AUROC value of 0.68 (95% CI, 0.64-0.72), and the augmented screening indicator had an AUROC value of 0.66 (95% CI, 0.63-0.70). Calibration was poor for both models but improved after recalibration.

Conclusion and Relevance

This prognostic study found that existing risk identification models for suicide prevention held promise when applied to new contexts and performed better than relying on a combined indictor of a positive suicide risk screen result, history of attempt, and recent suicidal ideation.

Indigenous data governance approaches applied in research using routinely collected health data: a scoping review

Globally, there is a growing acknowledgment of Indigenous Peoples' rights to control data related to their communities. This is seen in the development of Indigenous Data Governance standards. As health data collection increases, it's crucial to apply these standards in research involving Indigenous communities. Our study, therefore, aims to systematically review research using routinely collected health data of Indigenous Peoples, understanding the Indigenous Data Governance approaches and the associated advantages and challenges. We searched electronic databases for studies from 2013 to 2022, resulting in 85 selected articles. Of these, 65 (77%) involved Indigenous Peoples in the research, and 60 (71%) were authored by Indigenous individuals or organisations. While most studies (93%) provided ethical approval details, only 18 (21%) described Indigenous guiding principles, 35 (41%) reported on data sovereignty, and 28 (33%) addressed consent. This highlights the increasing focus on Indigenous Data Governance in utilising health data. Leveraging existing data sources in line with Indigenous data governance principles is vital for better understanding Indigenous health outcomes.

Predicting suicide death after emergency department visits with mental health or self-harm diagnoses

OBJECTIVE

Use health records data to predict suicide death following emergency department visits.

METHODS

Electronic health records and insurance claims from seven health systems were used to: identify emergency department visits with mental health or self-harm diagnoses by members aged 11 or older; extract approximately 2500 potential predictors including demographic, historical, and baseline clinical characteristics; and ascertain subsequent deaths by self-harm. Logistic regression with lasso and random forest models predicted self-harm death over 90 days after each visit.

RESULTS

Records identified 2,069,170 eligible visits, 899 followed by suicide death within 90 days. The best-fitting logistic regression with lasso model yielded an area under the receiver operating curve of 0.823 (95% CI 0.810-0.836). Visits above the 95th percentile of predicted risk included 34.8% (95% CI 31.1-38.7) of subsequent suicide deaths and had a 0.303% (95% CI 0.261-0.346) suicide death rate over the following 90 days. Model performance was similar across subgroups defined by age, sex, race, and ethnicity.

CONCLUSIONS

Machine learning models using coded data from health records have moderate performance in predicting suicide death following emergency department visits for mental health or self-harm diagnosis and could be used to identify patients needing more systematic follow-up.