Predicting suicide attempts and suicide deaths among adolescents following outpatient visits

Adolescent, Parent, and Provider Perceptions of a Predictive Algorithm to Identify Adolescent Suicide Risk in Primary Care

OBJECTIVE

To understand adolescent, parent, and provider perceptions of a machine learning algorithm for detecting adolescent suicide risk prior to its implementation primary care.

METHODS

We conducted semi-structured, qualitative interviews with adolescents (n = 9), parents (n = 12), and providers (n = 10; mixture of behavioral health and primary care providers) across two major health systems. Interviews were audio recorded and transcribed with analyses supported by use of NVivo. A codebook was developed combining codes derived inductively from interview transcripts and deductively from implementation science frameworks for content analysis.

RESULTS

Reactions to the algorithm were mixed. While many participants expressed privacy concerns, they believed the algorithm could be clinically useful for identifying adolescents at risk for suicide and facilitating follow-up. Parents' past experiences with their adolescents' suicidal thoughts and behaviors contributed to their openness to the algorithm. Results also aligned with several key Consolidated Framework for Implementation Research domains. For example, providers mentioned barriers inherent to the primary care setting such as time and resource constraints likely to impact algorithm implementation. Participants also cited a climate of mistrust of science and health care as potential barriers.

CONCLUSIONS

Findings shed light on factors that warrant consideration to promote successful implementation of suicide predictive algorithms in pediatric primary care. By attending to perspectives of potential end users prior to the development and testing of the algorithm, we can ensure that the risk prediction methods will be well-suited to the providers who would be interacting with them and the families who could benefit.

Prediction of non-suicidal self-injury in adolescents at the family level using regression methods and machine learning

BACKGROUND

Adolescent non-suicidal self-injury (NSSI) is a major public health issue. Family factors are significantly associated with NSSI in adolescents, while studies on forecasting NSSI at the family level are still limited. In addition to regression methods, machine learning (ML) techniques have been recommended to improve the accuracy of family-level risk prediction for NSSI.

METHODS

Using a dataset of 7967 students and their primary caregivers from a cross-sectional study, logistic regression model and random forest model were used to test the forecasting accuracy of NSSI predictions at the family level. Cross-validation was used to assess model prediction performance, including the area under the receiver operator curve (AUC), precision, Brier score, accuracy, sensitivity, specificity, positive predictive value and negative predictive value.

RESULTS

The top three important family-related predictors within the random forest algorithm included family function (importance:42.66), family conflict (importance:42.18), and parental depression (importance:27.21). The most significant family-related risk predictors and protective predictors identified by the logistic regression model were family history of mental illness (OR:2.25) and help-seeking behaviors of mental distress from parents (OR:0.65), respectively. The AUCs of the two models, logistic regression and random forest, were 0.852 and 0.835, respectively.

LIMITATIONS

The key limitation is that this cross-sectional survey only enabled the authors to examine predictors that were considered to be proximal rather than distal.

CONCLUSIONS

These findings highlight the significance of family-related factors in forecasting NSSI in adolescents. Combining both conventional statistical methods and ML methods to improve risk assessment of NSSI at the family level deserves attention.

Adolescents Who Do Not Endorse Risk via the Patient Health Questionnaire Before Self-Harm or Suicide

Importance

Given that the Patient Health Questionnaire (PHQ) item 9 is commonly used to screen for risk of self-harm and suicide, it is important that clinicians recognize circumstances when at-risk adolescents may go undetected.

Objective

To understand characteristics of adolescents with a history of depression who do not endorse the PHQ item 9 before a near-term intentional self-harm event or suicide.

Design, Setting, and Participants

This was a retrospective cohort study design using electronic health record and claims data from January 2009 through September 2017. Settings included primary care and mental health specialty clinics across 7 integrated US health care systems. Included in the study were adolescents aged 13 to 17 years with history of depression who completed the PHQ item 9 within 30 or 90 days before self-harm or suicide. Study data were analyzed September 2022 to April 2023.

Exposures

Demographic, diagnostic, treatment, and health care utilization characteristics.

Main Outcome(s) and Measure(s)

Responded "not at all" (score = 0) to PHQ item 9 regarding thoughts of death or self-harm within 30 or 90 days before self-harm or suicide.

Results

The study included 691 adolescents (mean [SD] age, 15.3 [1.3] years; 541 female [78.3%]) in the 30-day cohort and 1024 adolescents (mean [SD] age, 15.3 [1.3] years; 791 female [77.2%]) in the 90-day cohort. A total of 197 of 691 adolescents (29%) and 330 of 1024 adolescents (32%), respectively, scored 0 before self-harm or suicide on the PHQ item 9 in the 30- and 90-day cohorts. Adolescents seen in primary care (odds ratio [OR], 1.5; 95% CI, 1.0-2.1; P = .03) and older adolescents (OR, 1.2; 95% CI, 1.0-1.3; P = .02) had increased odds of scoring 0 within 90 days of a self-harm event or suicide, and adolescents with a history of inpatient hospitalization and a mental health diagnosis had twice the odds (OR, 2.0; 95% CI, 1.3-3.0; P = .001) of scoring 0 within 30 days. Conversely, adolescents with diagnoses of eating disorders were significantly less likely to score 0 on item 9 (OR, 0.4; 95% CI, 0.2-0.8; P = .007) within 90 days.

Conclusions and Relevance

Study results suggest that older age, history of an inpatient mental health encounter, or being screened in primary care were associated with at-risk adolescents being less likely to endorse having thoughts of death and self-harm on the PHQ item 9 before a self-harm event or suicide death. As use of the PHQ becomes more widespread in practice, additional research is needed for understanding reasons why many at-risk adolescents do not endorse thoughts of death and self-harm.

High dimensional predictions of suicide risk in 4.2 million US Veterans using ensemble transfer learning

We present an ensemble transfer learning method to predict suicide from Veterans Affairs (VA) electronic medical records (EMR). A diverse set of base models was trained to predict a binary outcome constructed from reported suicide, suicide attempt, and overdose diagnoses with varying choices of study design and prediction methodology. Each model used twenty cross-sectional and 190 longitudinal variables observed in eight time intervals covering 7.5 years prior to the time of prediction. Ensembles of seven base models were created and fine-tuned with ten variables expected to change with study design and outcome definition in order to predict suicide and combined outcome in a prospective cohort. The ensemble models achieved c-statistics of 0.73 on 2-year suicide risk and 0.83 on the combined outcome when predicting on a prospective cohort of [Formula: see text] 4.2 M veterans. The ensembles rely on nonlinear base models trained using a matched retrospective nested case-control (Rcc) study cohort and show good calibration across a diversity of subgroups, including risk strata, age, sex, race, and level of healthcare utilization. In addition, a linear Rcc base model provided a rich set of biological predictors, including indicators of suicide, substance use disorder, mental health diagnoses and treatments, hypoxia and vascular damage, and demographics.

Determining Distinct Suicide Attempts From Recurrent Electronic Health Record Codes: Classification Study

BACKGROUND

Prior suicide attempts are a relatively strong risk factor for future suicide attempts. There is growing interest in using longitudinal electronic health record (EHR) data to derive statistical risk prediction models for future suicide attempts and other suicidal behavior outcomes. However, model performance may be inflated by a largely unrecognized form of "data leakage" during model training: diagnostic codes for suicide attempt outcomes may refer to prior attempts that are also included in the model as predictors.

OBJECTIVE

We aimed to develop an automated rule for determining when documented suicide attempt diagnostic codes identify distinct suicide attempt events.

METHODS

From a large health care system's EHR, we randomly sampled suicide attempt codes for 300 patients with at least one pair of suicide attempt codes documented at least one but no more than 90 days apart. Supervised chart reviewers assigned the clinical settings (ie, emergency department [ED] versus non-ED), methods of suicide attempt, and intercode interval (number of days). The probability (or positive predictive value) that the second suicide attempt code in a given pair of codes referred to a distinct suicide attempt event from its preceding suicide attempt code was calculated by clinical setting, method, and intercode interval.

RESULTS

Of 1015 code pairs reviewed, 835 (82.3%) were nonindependent (ie, the 2 codes referred to the same suicide attempt event). When the second code in a pair was documented in a clinical setting other than the ED, it represented a distinct suicide attempt 3.3% of the time. The more time elapsed between codes, the more likely the second code in a pair referred to a distinct suicide attempt event from its preceding code. Code pairs in which the second suicide attempt code was assigned in an ED at least 5 days after its preceding suicide attempt code had a positive predictive value of 0.90.

CONCLUSIONS

EHR-based suicide risk prediction models that include International Classification of Diseases codes for prior suicide attempts as a predictor may be highly susceptible to bias due to data leakage in model training. We derived a simple rule to distinguish codes that reflect new, independent suicide attempts: suicide attempt codes documented in an ED setting at least 5 days after a preceding suicide attempt code can be confidently treated as new events in EHR-based suicide risk prediction models. This rule has the potential to minimize upward bias in model performance when prior suicide attempts are included as predictors in EHR-based suicide risk prediction models.

A systematic review of clinical health conditions predicted by machine learning diagnostic and prognostic models trained or validated using real-world primary health care data

With the advances in technology and data science, machine learning (ML) is being rapidly adopted by the health care sector. However, there is a lack of literature addressing the health conditions targeted by the ML prediction models within primary health care (PHC) to date. To fill this gap in knowledge, we conducted a systematic review following the PRISMA guidelines to identify health conditions targeted by ML in PHC. We searched the Cochrane Library, Web of Science, PubMed, Elsevier, BioRxiv, Association of Computing Machinery (ACM), and IEEE Xplore databases for studies published from January 1990 to January 2022. We included primary studies addressing ML diagnostic or prognostic predictive models that were supplied completely or partially by real-world PHC data. Studies selection, data extraction, and risk of bias assessment using the prediction model study risk of bias assessment tool were performed by two investigators. Health conditions were categorized according to international classification of diseases (ICD-10). Extracted data were analyzed quantitatively. We identified 106 studies investigating 42 health conditions. These studies included 207 ML prediction models supplied by the PHC data of 24.2 million participants from 19 countries. We found that 92.4% of the studies were retrospective and 77.3% of the studies reported diagnostic predictive ML models. A majority (76.4%) of all the studies were for models' development without conducting external validation. Risk of bias assessment revealed that 90.8% of the studies were of high or unclear risk of bias. The most frequently reported health conditions were diabetes mellitus (19.8%) and Alzheimer's disease (11.3%). Our study provides a summary on the presently available ML prediction models within PHC. We draw the attention of digital health policy makers, ML models developer, and health care professionals for more future interdisciplinary research collaboration in this regard.

Implementation of Suicide Risk Estimation Analytics to Support Mental Health Care for Quality Improvement

This quality improvement study describes use of estimation analytics to augment existing suicide prevention practices during routine mental health specialty encounters at a large US health care system.