Development and Validation of a Machine Learning Prediction Model of Posttraumatic Stress Disorder After Military Deployment

Estimating Treatment Effect Heterogeneity in Psychiatry: A Review and Tutorial With Causal Forests

BACKGROUND

Flexible machine learning tools are increasingly used to estimate heterogeneous treatment effects.

AIMS

This paper gives an accessible tutorial demonstrating the use of the causal forest algorithm, available in the R package grf.

SUMMARY

We start with a brief non-technical overview of treatment effect estimation methods, focusing on estimation in observational studies; the same techniques can also be applied in experimental studies. We then discuss the logic of estimating heterogeneous effects using the extension of the random forest algorithm implemented in grf. Finally, we illustrate causal forest by conducting a secondary analysis on the extent to which individual differences in resilience to high combat stress can be measured among US Army soldiers deploying to Afghanistan based on information about these soldiers available prior to deployment. We illustrate simple and interpretable exercises for model selection and evaluation, including targeting operator characteristics curves, Qini curves, area-under-the-curve summaries, and best linear projections.

RESULTS

A replication script with simulated data is available at https://github.com/grf-labs/grf/tree/master/experiments/ijmpr.

Machine learning algorithms for predicting PTSD: a systematic review and meta-analysis

This study aimed to compare and evaluate the prediction accuracy and risk of bias (ROB) of post-traumatic stress disorder (PTSD) predictive models. We conducted a systematic review and random-effect meta-analysis summarizing predictive model development and validation studies using machine learning in diverse samples to predict PTSD. Model performances were pooled using the area under the curve (AUC) with a 95% confidence interval (CI). Heterogeneity in each meta-analysis was measured using I2. The risk of bias in each study was appraised using the PROBAST tool. 48% of the 23 included studies had a high ROB, and the remaining had unclear. Tree-based models were the primarily used algorithms and showed promising results in predicting PTSD outcomes for various groups, as indicated by their pooled AUCs: military incidents (0.745), sexual or physical trauma (0.861), natural disasters (0.771), medical trauma (0.808), firefighters (0.96), and alcohol-related stress (0.935). However, the applicability of these findings is limited due to several factors, such as significant variability among the studies, high and unclear risks of bias, and a shortage of models that maintain accuracy when tested in new settings. Researchers should follow the reporting standards for AI/ML and adhere to the PROBAST guidelines. It is also essential to conduct external validations of these models to ensure they are practical and relevant in real-world settings.

BNC210, an α7 Nicotinic Receptor Modulator, in Post-Traumatic Stress Disorder

BACKGROUND

Post-traumatic stress disorder (PTSD) is a serious, debilitating, and prevalent psychiatric condition occurring in people who are traumatized and experience intense, disturbing thoughts and feelings that persist. BNC210 is a novel α7 nicotinic acetylcholine receptor-negative allosteric modulator developed to treat PTSD.

METHODS

ATTUNE was a randomized, double-blind, phase 2b, placebo-controlled trial. Patients between 18 and 75 years of age with a current PTSD diagnosis and a Clinician-Administered PTSD Scale for DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition) (CAPS-5) total symptom severity score of 30 or more were eligible (range: 0 to 80; in all scales used in this trial a higher score indicates a more severe condition). We randomly assigned patients 1:1 to a BNC210 dose of 900 mg twice daily or placebo for 12 weeks. The primary end point was a change from baseline to week 12 in CAPS-5 total score for BNC210 versus placebo.

RESULTS

In the modified intent-to-treat population (n=182), an improvement in the CAPS-5 total score was observed with BNC210 compared with placebo (least squares [LS] mean difference: -4.03; Cohen's d effect size: 0.40; P=0.048) at week 12. A LS mean difference in CAPS-5 score of -4.11 was observed as early as week 4. The LS mean difference to week 12 for depressive symptoms measured on the Montgomery-Åsberg Depression Rating Scale (range: 0 to 60; minimal clinically important difference [MCID] ≥2]) was -3.19 and for sleep measured on the Insomnia Severity Index (range: 0 to 28; MCID 6) was -2.19. Treatment-emergent adverse events (AEs) occurred in 70 (66.7%) patients in the BNC210 group and 56 (53.8%) in the placebo group, most commonly headache, nausea, fatigue, and hepatic enzyme elevations. In the BNC210 treatment group, 21 patients withdrew from treatment for AEs, while 10 did so in the placebo group. There were no serious AEs or deaths reported for the BNC210 group.

CONCLUSIONS

BNC210 improved PTSD symptom severity at week 12 with indications of effect as early as week 4. Our trial establishes equipoise for additional larger trials that are needed to determine the clinical utility of BNC210 for the treatment of PTSD. (Funded by Bionomics Limited; ClinicalTrials.gov number, NCT04951076.).

A systematic review of machine learning findings in PTSD and their relationships with theoretical models

In recent years, the application of machine learning (ML) techniques in research on the prediction of post-traumatic stress disorder (PTSD) has increased. However, concerns regarding the clinical relevance and generalizability of ML findings hamper their implementation by clinicians and researchers. Here in this systematic review we examined (1) the extent to which pre-, peri- and post-traumatic risk factors identified using ML approaches coincide with the theoretical understanding of the disorder; (2) whether new insights were gained through ML techniques; and (3) whether ML findings, combined with previous research, enable an integrative model of PTSD risk encompassing both predictor categories and their theoretical relevance. We reviewed ML studies on PTSD risk factors in PubMed, Web of Science and Scopus. Studies were included if they specified when predictors and PTSD symptoms were collected in temporal relation to the traumatic event. A total of 30 studies with 12,908 participants (mean age 36.5 years) were included. After extracting the 15 most important predictors from all studies, we categorized them into pre-, peri- and post-trauma exposure predictors and examined their associations with established theoretical models of PTSD. Many studies exhibited a risk of bias, assessed using the prediction model risk of bias assessment tool (PROBAST). However, we found overlaps in identified predictors across studies, a concordance between data-driven results and theory-driven research, and underexplored predictors identified through ML. We propose an integrative model of PTSD risk that incorporates both data-driven and theory-driven findings and discuss future directions. We emphasize the importance of standards on how to apply and report ML approaches for mental health.

Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study

BACKGROUND

The performance of a classification algorithm eventually reaches a point of diminishing returns, where the additional sample added does not improve the results. Thus, there is a need to determine an optimal sample size that maximizes performance while accounting for computational burden or budgetary concerns.

OBJECTIVE

This study aimed to determine optimal sample sizes and the relationships between sample size and dataset-level characteristics over a variety of binary classification algorithms.

METHODS

A total of 16 large open-source datasets were collected, each containing a binary clinical outcome. Furthermore, 4 machine learning algorithms were assessed: XGBoost (XGB), random forest (RF), logistic regression (LR), and neural networks (NNs). For each dataset, the cross-validated area under the curve (AUC) was calculated at increasing sample sizes, and learning curves were fit. Sample sizes needed to reach the observed full-dataset AUC minus 2 points (0.02) were calculated from the fitted learning curves and compared across the datasets and algorithms. Dataset-level characteristics, minority class proportion, full-dataset AUC, number of features, type of features, and degree of nonlinearity were examined. Negative binomial regression models were used to quantify relationships between these characteristics and expected sample sizes within each algorithm. A total of 4 multivariable models were constructed, which selected the best-fitting combination of dataset-level characteristics.

RESULTS

Among the 16 datasets (full-dataset sample sizes ranging from 70,000-1,000,000), median sample sizes were 9960 (XGB), 3404 (RF), 696 (LR), and 12,298 (NN) to reach AUC stability. For all 4 algorithms, more balanced classes (multiplier: 0.93-0.96 for a 1% increase in minority class proportion) were associated with decreased sample size. Other characteristics varied in importance across algorithms-in general, more features, weaker features, and more complex relationships between the predictors and the response increased expected sample sizes. In multivariable analysis, the top selected predictors were minority class proportion among all 4 algorithms assessed, full-dataset AUC (XGB, RF, and NN), and dataset nonlinearity (XGB, RF, and NN). For LR, the top predictors were minority class proportion, percentage of strong linear features, and number of features. Final multivariable sample size models had high goodness-of-fit, with dataset-level predictors explaining a majority (66.5%-84.5%) of the total deviance in the data among all 4 models.

CONCLUSIONS

The sample sizes needed to reach AUC stability among 4 popular classification algorithms vary by dataset and method and are associated with dataset-level characteristics that can be influenced or estimated before the start of a research study.

A prediction model for differential resilience to the effects of combat-related stressors in US army soldiers

OBJECTIVES

To develop a composite score for differential resilience to effects of combat-related stressors (CRS) on persistent DSM-IV post-traumatic stress disorder (PTSD) among US Army combat arms soldiers using survey data collected before deployment.

METHODS

A sample of n = 2542 US Army combat arms soldiers completed a survey shortly before deployment to Afghanistan and then again two to three and 8-9 months after redeployment. Retrospective self-reports were obtained about CRS. Precision treatment methods were used to determine whether differential resilience to persistent PTSD in the follow-up surveys could be developed from pre-deployment survey data in a 60% training sample and validated in a 40% test sample.

RESULTS

40.8% of respondents experienced high CRS and 5.4% developed persistent PTSD. Significant test sample heterogeneity was found in resilience (t = 2.1, p = 0.032), with average treatment effect (ATE) of high CRS in the 20% least resilient soldiers of 17.1% (SE = 5.5%) compared to ATE = 3.8% (SE = 1.2%) in the remaining 80%. The most important predictors involved recent and lifetime pre-deployment distress disorders.

CONCLUSIONS

A reliable pre-deployment resilience score can be constructed to predict variation in the effects of high CRS on persistent PTSD among combat arms soldiers. Such a score could be used to target preventive interventions to reduce PTSD or other resilience-related outcomes.

Diagnostic Value of Inflammatory Biomarkers in Intracranial Venous Thrombosis: A Multi-model Predictive Analysis

OBJECTIVE

Cerebral venous thrombosis (CVT) is a rare but significant condition, primarily affecting young adults, especially women. The diagnosis of CVT is challenging due to its nonspecific clinical presentation. Inflammatory biomarkers, such as the systemic immune-inflammatory index (SII), platelet-to-lymphocyte ratio (PLR), and neutrophil-to-lymphocyte ratio (NLR), may aid in early diagnosis. This study aimed to explore the role of these biomarkers and assess machine learning models for improving diagnostic accuracy.

METHODS

This study included 100 CVT patients and 50 controls. Data collected included demographic information, biochemical markers, and clinical symptoms. Traditional statistical methods and machine learning models, including decision trees, random forests, AdaBoost, k-nearest neighbors, support vector machines (SVM), and artificial neural networks (ANN), were used to evaluate the diagnostic value of biomarkers.

RESULTS

The SII and NLR levels were significantly higher in CVT patients. The ANN model based on SII and PLR achieved the best diagnostic performance, with an area under the curve (AUC) of 0.94, showing high accuracy and reliability.

CONCLUSION

Inflammatory biomarkers, particularly SII, have significant predictive value in CVT diagnosis. Machine learning models, especially ANN, show promise in improving diagnostic accuracy. Future studies with larger sample sizes are needed to validate these findings further.

Deciphering the environmental chemical basis of muscle quality decline by interpretable machine learning models

BACKGROUND

Sarcopenia is known as a decline in skeletal muscle quality and function that is associated with age. Sarcopenia is linked to diverse health problems, including endocrine-related diseases. Environmental chemicals (ECs), a broad class of chemicals released from industry, may influence muscle quality decline.

OBJECTIVES

In this work, we aimed to simultaneously elucidate the associations between muscle quality decline and diverse EC exposures based on the data from the 2011-2012 and 2013-2014 survey cycles in the National Health and Nutrition Examination Survey (NHANES) project using machine learning models.

METHODS

Six machine learning models were trained based on the EC and non-EC exposures from NHANES to distinguish low from normal muscle quality index status. Different machine learning metrics were evaluated for these models. The Shapley additive explanations (SHAP) approach was used to provide explainability for machine learning models.

RESULTS

Random forest (RF) performed best on the independent testing data set. Based on the testing data set, ECs can independently predict the binary muscle quality status with good performance by RF (area under the receiver operating characteristic curve = 0.793; area under the precision-recall curve = 0.808). The SHAP ranked the importance of ECs for the RF model. As a result, several metals and chemicals in urine, including 3-phenoxybenzoic acid and cobalt, were more associated with the muscle quality decline.

CONCLUSIONS

Altogether, our analyses suggest that ECs can independently predict muscle quality decline with a good performance by RF, and the SHAP-identified ECs can be closely related to muscle quality decline and sarcopenia. Our analyses may provide valuable insights into ECs that may be the important basis of sarcopenia and endocrine-related diseases in United States populations.

Parvalbumin interneuron activity in autism underlies susceptibility to PTSD-like memory formation

A rising concern in autism spectrum disorder (ASD) is the heightened sensitivity to trauma, the potential consequences of which have been overlooked, particularly upon the severity of the ASD traits. We first demonstrate a reciprocal relationship between ASD and post-traumatic stress disorder (PTSD) and reveal that exposure to a mildly stressful event induces PTSD-like memory in four mouse models of ASD. We also establish an unanticipated consequence of stress, as the formation of PTSD-like memory leads to the aggravation of core autistic traits. Such a susceptibility to developing PTSD-like memory in ASD stems from hyperactivation of the prefrontal cortex and altered fine-tuning of parvalbumin interneuron firing. Traumatic memory can be treated by recontextualization, reducing the deleterious effects on the core symptoms of ASD in the Cntnap2 KO mouse model. This study provides a neurobiological and psychological framework for future examination of the impact of PTSD-like memory in autism.

Novel multiclass classification machine learning approach for the early-stage classification of systemic autoimmune rheumatic diseases

OBJECTIVE

Systemic autoimmune rheumatic diseases (SARDs) encompass a diverse group of complex conditions with overlapping clinical features, making accurate diagnosis challenging. This study aims to develop a multiclass machine learning (ML) model for early-stage SARDs classification using accessible laboratory indicators.

METHODS

A total of 925 SARDs patients were included, categorised into SLE, Sjögren's syndrome (SS) and inflammatory myositis (IM). Clinical characteristics and laboratory markers were collected and nine key indicators, including anti-dsDNA, anti-SS-A60, anti-Sm/nRNP, antichromatin, anti-dsDNA (indirect immunofluorescence assay), haemoglobin (Hb), platelet, neutrophil percentage and cytoplasmic patterns (AC-19, AC-20), were selected for model building. Various ML algorithms were used to construct a tripartite classification ML model.

RESULTS

Patients were divided into two cohorts, cohort 1 was used to construct a tripartite classification model. Among models assessed, the random forest (RF) model demonstrated superior performance in distinguishing SLE, IM and SS (with area under curve=0.953, 0.903 and 0.836; accuracy= 0.892, 0.869 and 0.857; sensitivity= 0.890, 0.868 and 0.795; specificity= 0.910, 0.836 and 0.748; positive predictive value=0.922, 0.727 and 0.663; and negative predictive value= 0.854, 0.915 and 0.879). The RF model excelled in classifying SLE (precision=0.930, recall=0.985, F1 score=0.957). For IM and SS, RF model outcomes were (precision=0.793, 0.950; recall=0.920, 0.679; F1 score=0.852, 0.792). Cohort 2 served as an external validation set, achieving an overall accuracy of 87.3%. Individual classification performances for SLE, SS and IM were excellent, with precision, recall and F1 scores specified. SHAP analysis highlighted significant contributions from antibody profiles.

CONCLUSION

This pioneering multiclass ML model, using basic laboratory indicators, enhances clinical feasibility and demonstrates promising potential for SARDs classification. The collaboration of clinical expertise and ML offers a nuanced approach to SARDs classification, with potential for enhanced patient care.

Development and validation of a machine learning model using electronic health records to predict trauma- and stressor-related psychiatric disorders after hospitalization with sepsis

A significant minority of individuals develop trauma- and stressor-related disorders (TSRD) after surviving sepsis, a life-threatening immune response to infections. Accurate prediction of risk for TSRD can facilitate targeted early intervention strategies, but many existing models rely on research measures that are impractical to incorporate to standard emergency department workflows. To increase the feasibility of implementation, we developed models that predict TSRD in the year after survival from sepsis using only electronic health records from the hospitalization (n = 217,122 hospitalizations from 2012-2015). The optimal model was evaluated in a temporally independent prospective test sample (n = 128,783 hospitalizations from 2016-2017), where patients in the highest-risk decile accounted for nearly one-third of TSRD cases. Our approach demonstrates that risk for TSRD after sepsis can be stratified without additional assessment burden on clinicians and patients, which increases the likelihood of model implementation in hospital settings.