Assessment of a Prediction Model for Antidepressant Treatment Stability Using Supervised Topic Models

Recommendations for defining treatment outcomes in major psychiatric disorders using real-world data

Although information from real-world data can be used to identify factors that aid treatment choice, there are no guidelines for the use of such data. The aim of this Review is to summarise and evaluate definitions of treatment outcomes for antidepressants, antipsychotics, and mood stabilisers when using real-world data, and to suggest standards for the field. Given that no standards for the use of these data in estimating treatment outcomes exist, variability is high for treatment outcome definitions. We make recommendations for different scenarios of available data and highlight the importance of using other sources of information to validate proxy measures such as continued treatment, switching between medications, or polypharmacy of psychotropic medications. Well defined and validated treatment outcome measures that incorporate real-world data could facilitate the development of precision psychiatry approaches and support regulatory decision making regarding psychopharmacological agents.

Heterogeneity in Antidepressant Treatment and Major Depressive Disorder Outcomes Among Clinicians

Importance

While abundant work has examined patient-level differences in antidepressant treatment outcomes, little is known about the extent of clinician-level differences. Understanding these differences may be important in the development of risk models, precision treatment strategies, and more efficient systems of care.

Objective

To characterize differences between outpatient clinicians in treatment selection and outcomes for their patients diagnosed with major depressive disorder across academic medical centers, community hospitals, and affiliated clinics.

Design, Setting, and Participants

This was a longitudinal cohort study using data derived from electronic health records at 2 large academic medical centers and 6 community hospitals, and their affiliated outpatient networks, in eastern Massachusetts. Participants were deidentified clinicians who billed at least 10 International Classification of Diseases, Ninth Revision (ICD-9) or Tenth Revision (ICD-10) diagnoses of major depressive disorder per year between 2008 and 2022. Data analysis occurred between September 2023 and January 2024.

Main Outcomes and Measures

Heterogeneity of prescribing, defined as the number of distinct antidepressants accounting for 75% of prescriptions by a given clinician; proportion of patients who did not return for follow-up after an index prescription; and proportion of patients receiving stable, ongoing antidepressant treatment.

Results

Among 11 934 clinicians treating major depressive disorder, unsupervised learning identified 10 distinct clusters on the basis of ICD codes, corresponding to outpatient psychiatry as well as oncology, obstetrics, and primary care. Between these clusters, substantial variability was identified in the proportion of selective serotonin reuptake inhibitors, selective norepinephrine reuptake inhibitors, and tricyclic antidepressants prescribed, as well as in the number of distinct antidepressants prescribed. Variability was also detected between clinician clusters in loss to follow-up and achievement of stable treatment, with the former ranging from 27% to 69% and the latter from 22% to 42%. Clinician clusters were significantly associated with treatment outcomes.

Conclusions and Relevance

Groups of clinicians treating individuals diagnosed with major depressive disorder exhibit marked differences in prescribing patterns as well as longitudinal patient outcomes defined by electronic health records. Incorporating these group identifiers yielded similar prediction to more complex models incorporating individual codes, suggesting the importance of considering treatment context in efforts at risk stratification.

Optimizing precision medicine for second-step depression treatment: a machine learning approach

BACKGROUND

Less than a third of patients with depression achieve successful remission with standard first-step antidepressant monotherapy. The process for determining appropriate second-step care is often based on clinical intuition and involves a protracted course of trial and error, resulting in substantial patient burden and unnecessary delay in the provision of optimal treatment. To address this problem, we adopt an ensemble machine learning approach to improve prediction accuracy of remission in response to second-step treatments.

METHOD

Data were derived from the Level 2 stage of the STAR*D dataset, which included 1439 patients who were randomized into one of seven different second-step treatment strategies after failing to achieve remission during first-step antidepressant treatment. Ensemble machine learning models, comprising several individual algorithms, were evaluated using nested cross-validation on 155 predictor variables including clinical and demographic measures.

RESULTS

The ensemble machine learning algorithms exhibited differential classification performance in predicting remission status across the seven second-step treatments. For the full set of predictors, AUC values ranged from 0.51 to 0.82 depending on the second-step treatment type. Predicting remission was most successful for cognitive therapy (AUC = 0.82) and least successful for other medication and combined treatment options (AUCs = 0.51-0.66).

CONCLUSION

Ensemble machine learning has potential to predict second-step treatment. In this study, predictive performance varied by type of treatment, with greater accuracy in predicting remission in response to behavioral treatments than to pharmacotherapy interventions. Future directions include considering more informative predictor modalities to enhance prediction of second-step treatment response.

Personalized anti-tumor drug efficacy prediction based on clinical data

Anti-tumor drug efficacy prediction poses an unprecedented challenge to realizing personalized medicine. This paper proposes to predict personalized anti-tumor drug efficacy based on clinical data. Specifically, we encode the clinical text as numeric vectors featured with hidden topics for patients using Latent Dirichlet Allocation model. Then, to classify patients into two classes, responsive or non-responsive to a drug, drug efficacy predictors are established by machine learning based on the Latent Dirichlet Allocation topic representation. To evaluate the proposed method, we collected and collated clinical records of lung and bowel cancer patients treated with platinum. Experimental results on the data sets show the efficacy and effectiveness of the proposed method, suggesting the potential value of clinical data in cancer precision medicine. We hope that it will promote the research of drug efficacy prediction based on clinical data.

Electronic health records and stratified psychiatry: bridge to precision treatment?

The use of a stratified psychiatry approach that combines electronic health records (EHR) data with machine learning (ML) is one potentially fruitful path toward rapidly improving precision treatment in clinical practice. This strategy, however, requires confronting pervasive methodological flaws as well as deficiencies in transparency and reporting in the current conduct of ML-based studies for treatment prediction. EHR data shares many of the same data quality issues as other types of data used in ML prediction, plus some unique challenges. To fully leverage EHR data's power for patient stratification, increased attention to data quality and collection of patient-reported outcome data is needed.

Machine learning prediction will be part of future treatment of depression

Machine learning (ML) is changing the way that medicine is practiced. While already clinically utilised in diagnostic radiology and outcome prediction in intensive care unit, ML approaches in psychiatry remain nascent. Implementing ML algorithms in psychiatry, particularly in the treatment of depression, is significantly more challenging than other areas of medicine in part because of the less demarcated disease nosology and greater variability in practice. Given the current exiguous capacity of clinicians to predict patient and treatment outcomes in depression, there is a significantly greater need for better predictive capability. Early studies have shown promising results. ML predictions were significantly better than chance within the sequenced treatment alternatives to relieve depression (STAR*D) trial (accuracy 64.6%, p < 0.0001) and combining medications to enhance depression outcomes (COMED) randomised Controlled Trial (RCT) (accuracy 59.6%, p = 0.043), with similar results found in larger scale, retrospective studies. The greater flexibility and dimensionality of ML approaches has been demonstrated in studies incorporating diverse input variables including electroencephalography scans, achieving 88% accuracy for treatment response, and cognitive test scores, achieving up to 72% accuracy for treatment response. The predicting response to depression treatment (PReDicT) trial tested ML informed prescribing of antidepressants against standard therapy and found there was both better outcomes for anxiety and functional endpoints despite the algorithm only having a balanced accuracy of 57.5%. Impeding the progress of ML algorithms in psychiatry are pragmatic hurdles, including accuracy, expense, acceptability and comprehensibility, and ethical hurdles, including medicolegal liability, clinical autonomy and data privacy. Notwithstanding impediments, it is clear that ML prediction algorithms will be part of depression treatment in the future and clinicians should be prepared for their arrival.

Establishment of a Public Mental Health Database for Research Purposes in the Ferrara Province: Development and Preliminary Evaluation Study

Background

The immediate use of data exported from electronic health records (EHRs) for research is often limited by the necessity to transform data elements into an actual data set.

Objective

This paper describes the methodology for establishing a data set that originated from an EHR registry that included clinical, health service, and sociodemographic information.

Methods

The Extract, Transform, Load process was applied to raw data collected at the Integrated Department of Mental Health and Pathological Addictions in Ferrara, Italy, from 1925 to February 18, 2021, to build the new, anonymized Ferrara-Psychiatry (FEPSY) database. Information collected before the first EHR was implemented (ie, in 1991) was excluded. An unsupervised cluster analysis was performed to identify patient subgroups to support the proof of concept.

Results

The FEPSY database included 3,861,432 records on 46,222 patients. Since 1991, each year, a median of 1404 (IQR 1117.5-1757.7) patients had newly accessed care, and a median of 7300 (IQR 6109.5-9397.5) patients were actively receiving care. Among 38,022 patients with a mental disorder, 2 clusters were identified; the first predominantly included male patients who were aged 25 to 34 years at first presentation and were living with their parents, and the second predominantly included female patients who were aged 35 to 44 years and were living with their own families.

Conclusions

The process for building the FEPSY database proved to be robust and replicable with similar health care data, even when they were not originally conceived for research purposes. The FEPSY database will enable future in-depth analyses regarding the epidemiology and social determinants of mental disorders, access to mental health care, and resource utilization.

AI-assisted prediction of differential response to antidepressant classes using electronic health records

Antidepressant selection is largely a trial-and-error process. We used electronic health record (EHR) data and artificial intelligence (AI) to predict response to four antidepressants classes (SSRI, SNRI, bupropion, and mirtazapine) 4 to 12 weeks after antidepressant initiation. The final data set comprised 17,556 patients. Predictors were derived from both structured and unstructured EHR data and models accounted for features predictive of treatment selection to minimize confounding by indication. Outcome labels were derived through expert chart review and AI-automated imputation. Regularized generalized linear model (GLM), random forest, gradient boosting machine (GBM), and deep neural network (DNN) models were trained and their performance compared. Predictor importance scores were derived using SHapley Additive exPlanations (SHAP). All models demonstrated similarly good prediction performance (AUROCs ≥ 0.70, AUPRCs ≥ 0.68). The models can estimate differential treatment response probabilities both between patients and between antidepressant classes for the same patient. In addition, patient-specific factors driving response probabilities for each antidepressant class can be generated. We show that antidepressant response can be accurately predicted from real-world EHR data with AI modeling, and our approach could inform further development of clinical decision support systems for more effective treatment selection.

Research Letter: Application of GPT-4 to select next-step antidepressant treatment in major depression

Introduction

Large language models perform well on a range of academic tasks including medical examinations. The performance of this class of models in psychopharmacology has not been explored.

Method

Chat GPT-plus, implementing the GPT-4 large language model, was presented with each of 10 previously-studied antidepressant prescribing vignettes in randomized order, with results regenerated 5 times to evaluate stability of responses. Results were compared to expert consensus.

Results

At least one of the optimal medication choices was included among the best choices in 38/50 (76%) vignettes: 5/5 for 7 vignettes, 3/5 for 1, and 0/5 for 2. At least one of the poor choice or contraindicated medications was included among the choices considered optimal or good in 24/50 (48%) of vignettes. The model provided as rationale for treatment selection multiple heuristics including avoiding prior unsuccessful medications, avoiding adverse effects based on comorbidities, and generalizing within medication class.

Conclusion

The model appeared to identify and apply a number of heuristics commonly applied in psychopharmacologic clinical practice. However, the inclusion of less optimal recommendations indicates that large language models may pose a substantial risk if routinely applied to guide psychopharmacologic treatment without further monitoring.

The promise of machine learning in predicting treatment outcomes in psychiatry

For many years, psychiatrists have tried to understand factors involved in response to medications or psychotherapies, in order to personalize their treatment choices. There is now a broad and growing interest in the idea that we can develop models to personalize treatment decisions using new statistical approaches from the field of machine learning and applying them to larger volumes of data. In this pursuit, there has been a paradigm shift away from experimental studies to confirm or refute specific hypotheses towards a focus on the overall explanatory power of a predictive model when tested on new, unseen datasets. In this paper, we review key studies using machine learning to predict treatment outcomes in psychiatry, ranging from medications and psychotherapies to digital interventions and neurobiological treatments. Next, we focus on some new sources of data that are being used for the development of predictive models based on machine learning, such as electronic health records, smartphone and social media data, and on the potential utility of data from genetics, electrophysiology, neuroimaging and cognitive testing. Finally, we discuss how far the field has come towards implementing prediction tools in real-world clinical practice. Relatively few retrospective studies to-date include appropriate external validation procedures, and there are even fewer prospective studies testing the clinical feasibility and effectiveness of predictive models. Applications of machine learning in psychiatry face some of the same ethical challenges posed by these techniques in other areas of medicine or computer science, which we discuss here. In short, machine learning is a nascent but important approach to improve the effectiveness of mental health care, and several prospective clinical studies suggest that it may be working already.

Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise As Much As Possible

We propose Preferential MoE, a novel human-ML mixture-of-experts model that augments human expertise in decision making with a data-based classifier only when necessary for predictive performance. Our model exhibits an interpretable gating function that provides information on when human rules should be followed or avoided. The gating function is maximized for using human-based rules, and classification errors are minimized. We propose solving a coupled multi-objective problem with convex subproblems. We develop approximate algorithms and study their performance and convergence. Finally, we demonstrate the utility of Preferential MoE on two clinical applications for the treatment of Human Immunodeficiency Virus (HIV) and management of Major Depressive Disorder (MDD).

Visualizing Patterns of Medication Switching Among Major Depressive Patients with Various Stability and Difficulty to Treatments

INTRODUCTION

Efforts have been made in assessing efficacy and tolerability to various antidepressants, but understanding personalized chances of stability to medication switching sequence is still inconclusive. This study aimed to identify naturalistic switching patterns of medication in stratifying MDD patients.

METHODS

MDD patients were stratified based on treatment difficulty evaluated with the "Treatment Resistance to Antidepressants Evaluation Scale for Unipolar Depression" (TRADES). The duration of the time of diagnoses until the final switch to another class of antidepressants was used as prediction of unstable to drug therapy. ROC analysis was used to determine the cutoff values. A continuous temporal events function from the visual analytic tool was employed to perform patterns of switching between distinct pharmacological class such as selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine reuptake inhibitors (SNRIs).

RESULTS

TRADES scores of 4.5 and not-switching times of 12.5 months were used as cutoff values to divide patients into four subgroups: stable/easy-to-treat (SE), unstable/easy-to-treat (UE), stable/difficult-to-treat (SD) and unstable/difficult-to-treat (UD). A total of 80% and 76.9% of patients initially treated with the SSRIs paroxetine or fluoxetine, respectively, were predicted to be stable to drug therapy. Approximately 70%, 44.8% and 41.4% of patients initially treated with the SNRIs fluvoxamine, sertraline and venlafaxine, respectively, were predicted to be UD, and 60% of patients using duloxetine were predicted to be stable to drug therapy. Analysis of the switching phenomenon showed that SSRIs were the first prescribed medications and mostly taken by the stable subgroups, and SNRIs were the preferentially chosen switching alternative. Medication switching patterns in unstable MDD patients are discussed.

CONCLUSION

Paroxetine, fluoxetine and duloxetine users were mostly stable among MDD patients in Taiwan with various stability and difficulty to treatments. Although responsiveness to specific medication sequence is likely required for clinical application, the results provide a baseline for such studies.

Machine Learning Prediction of Treatment Outcome in Late-Life Depression

Background: Recent evidence suggests that integration of multi-modal data improves performance in machine learning prediction of depression treatment outcomes. Here, we compared the predictive performance of three machine learning classifiers using differing combinations of sociodemographic characteristics, baseline clinical self-reports, cognitive tests, and structural magnetic resonance imaging (MRI) features to predict treatment outcomes in late-life depression (LLD). Methods: Data were combined from two clinical trials conducted with depressed adults aged 60 and older, including response to escitalopram (N = 32, NCT01902004) and Tai Chi (N = 35, NCT02460666). Remission was defined as a score of 6 or less on the 24-item Hamilton Rating Scale for Depression (HAMD) at the end of 24 weeks of treatment. Features subsets were constructed from baseline sociodemographic and clinical features, gray matter volumes (GMVs), or both. Three classification algorithms were compared: (1) Support Vector Machine-Radial Bias Function (SVMRBF), (2) Random Forest (RF), and (3) Logistic Regression (LR). A repeated 5-fold cross-validation approach with a wrapper-based feature selection method was used for model fitting. Model performance metrics included Area under the ROC Curve (AUC) and Matthews correlation coefficient (MCC). Cross-validated performance significance was tested by permutation analysis. Classifiers were compared by Cochran's Q and post-hoc pairwise comparisons using McNemar's Chi-Square test with Bonferroni correction. Results: For the RF and SVMRBF algorithms, the combined feature set outperformed the clinical and GMV feature sets with a final cross-validated AUC of 0.83 ± 0.11 and 0.80 ± 0.11, respectively. Both classifiers passed permutation analysis. The LR algorithm performed best using GMV features alone (AUC 0.79 ± 0.14) but failed to pass permutation analysis using any feature set. Performance of the three classifiers differed significantly for all three features sets. Important predictive features of treatment response included anterior and posterior cingulate volumes, depression characteristics, and self-reported health-related quality scores. Conclusion: This preliminary exploration into the use of ML and multi-modal data to identify predictors of general treatment response in LLD indicates that integration of clinical and structural MRI features significantly increases predictive capability. Identified features are among those previously implicated in geriatric depression, encouraging future work in this arena.