A comparison of machine learning methods for classification using simulation with multiple real data examples from mental health studies

Investigating Protective and Risk Factors and Predictive Insights for Aboriginal Perinatal Mental Health: Explainable Artificial Intelligence Approach

BACKGROUND

Perinatal depression and anxiety significantly impact maternal and infant health, potentially leading to severe outcomes like preterm birth and suicide. Aboriginal women, despite their resilience, face elevated risks due to the long-term effects of colonization and cultural disruption. The Baby Coming You Ready (BCYR) model of care, centered on a digitized, holistic, strengths-based assessment, was co-designed to address these challenges. The successful BCYR pilot demonstrated its ability to replace traditional risk-based screens. However, some health professionals still overrely on psychological risk scores, often overlooking the contextual circumstances of Aboriginal mothers, their cultural strengths, and mitigating protective factors. This highlights the need for new tools to improve clinical decision-making.

OBJECTIVE

We explored different explainable artificial intelligence (XAI)-powered machine learning techniques for developing culturally informed, strengths-based predictive modeling of perinatal psychological distress among Aboriginal mothers. The model identifies and evaluates influential protective and risk factors while offering transparent explanations for AI-driven decisions.

METHODS

We used deidentified data from 293 Aboriginal mothers who participated in the BCYR program between September 2021 and June 2023 at 6 health care services in Perth and regional Western Australia. The original dataset includes variables spanning cultural strengths, protective factors, life events, worries, relationships, childhood experiences, family and domestic violence, and substance use. After applying feature selection and expert input, 20 variables were chosen as predictors. The Kessler-5 scale was used as an indicator of perinatal psychological distress. Several machine learning models, including random forest (RF), CatBoost (CB), light gradient-boosting machine (LightGBM), extreme gradient boosting (XGBoost), k-nearest neighbor (KNN), support vector machine (SVM), and explainable boosting machine (EBM), were developed and compared for predictive performance. To make the black-box model interpretable, post hoc explanation techniques including Shapley additive explanations and local interpretable model-agnostic explanations were applied.

RESULTS

The EBM outperformed other models (accuracy=0.849, 95% CI 0.8170-0.8814; F1-score=0.771, 95% CI 0.7169-0.8245; area under the curve=0.821, 95% CI 0.7829-0.8593) followed by RF (accuracy=0.829, 95% CI 0.7960-0.8617; F1-score=0.736, 95% CI 0.6859-0.7851; area under the curve=0.795, 95% CI 0.7581-0.8318). Explanations from EBM, Shapley additive explanations, and local interpretable model-agnostic explanations identified consistent patterns of key influential factors, including questions related to "Feeling Lonely," "Blaming Herself," "Makes Family Proud," "Life Not Worth Living," and "Managing Day-to-Day." At the individual level, where responses are highly personal, these XAI techniques provided case-specific insights through visual representations, distinguishing between protective and risk factors and illustrating their impact on predictions.

CONCLUSIONS

This study shows the potential of XAI-driven models to predict psychological distress in Aboriginal mothers and provide clear, human-interpretable explanations of how important factors interact and influence outcomes. These models may help health professionals make more informed, non-biased decisions in Aboriginal perinatal mental health screenings.

Individualized multi-modal MRI biomarkers predict 1-year clinical outcome in first-episode drug-naïve schizophrenia patients

Background

Antipsychotic medications offer limited long-term benefit to about 30% of patients with schizophrenia. We aimed to explore the individual-specific imaging markers to predict 1-year treatment response of schizophrenia.

Methods

Structural morphology and functional topological features related to treatment response were identified using an individualized parcellation analysis in conjunction with machine learning (ML). We performed dimensionality reductions using the Pearson correlation coefficient and three feature selection analyses and classifications using 10 ML classifiers. The results were assessed through a 5-fold cross-validation (training and validation cohorts, n = 51) and validated using the external test cohort (n = 17).

Results

ML algorithms based on individual-specific brain network proved more effective than those based on group-level brain network in predicting outcomes. The most predictive features based on individual-specific parcellation involved the GMV of the default network and the degree of the control, limbic, and default networks. The AUCs for the training, validation, and test cohorts were 0.947, 0.939, and 0.883, respectively. Additionally, the prediction performance of the models constructed by the different feature selection methods and classifiers showed no significant differences.

Conclusion

Our study highlighted the potential of individual-specific network parcellation in treatment resistant schizophrenia prediction and underscored the crucial role of feature attributes in predictive model accuracy.

Fingerprinting hyperglycemia using predictive modelling approach based on low-cost routine CBC and CRP diagnostics

Hyperglycemia is an outcome of dysregulated glucose homeostasis in the human body and may induce chronic elevation of blood glucose levels. Lifestyle factors such as overnutrition, physical inactivity, and psychosocials coupled with systemic low-grade inflammation have a strong negative impact on glucose homeostasis, in particular, insulin sensitivity. Together, these factors contribute to the pathophysiology of diabetes (DM) and expanding landscape of its prevalence regionally and globally. The rapid rise in the prevalence of type 2 diabetes, therefore, underscores the need for its early diagnosis and treatment. In this work, we have evaluated the discriminatory capacity of different diagnostic markers including inflammatory biomolecules and RBC (Red Blood Cell) indices in predicting the risk of hyperglycemia and borderline hyperglycemia. For that, 208,137 clinical diagnostic entries obtained over five years from Chugtai Labs, Pakistan, were retrospectively evaluated. The dataset included HbA1c (n = 142,011), complete blood count (CBC, n = 84,263), fasting blood glucose (FBG, n = 35,363), and C-reactive protein (CRP, n = 9035) tests. Our results provide four glycemic predictive models for two cohorts HbA1c and FBG) each having an overall predictive accuracy of more than 80% (p-value < 0.0001). Next, multivariate analysis (MANOVA) followed by univariate analysis (ANOVA) was employed to identify predictors with significant discriminatory capacity for different levels of glycemia. We show that the interplay between inflammation, hyperglycemic-induced derangements in RBC indices, and altered glucose homeostasis could be employed for prognosticating hyperglycemic outcomes. Our results then conclude a glycemic predictor with high sensitivity and specificity, employing inflammatory markers coupled with RBC indices, to predict glycemic outcomes (ROC p-value < 0.0001). Taken together, this study outlines a predictor of glycemic outcomes which could assist as a prophylactic intervention in predicting the early onset of hyperglycemia and borderline hyperglycemia.

Heart and brain traumatic stress biomarker analysis with and without machine learning: A scoping review

The enigma of post-traumatic stress disorder (PTSD) is embedded in a complex array of physiological responses to stressful situations that result in disruptions in arousal and cognitions that characterise the psychological disorder. Deciphering these physiological patterns is complex, which has seen the use of machine learning (ML) grow in popularity. However, it is unclear to what extent ML has been used with physiological data, specifically, the electroencephalogram (EEG) and electrocardiogram (ECG) to further understand the physiological responses associated with PTSD. To better understand the use of EEG and ECG biomarkers, with and without ML, a scoping review was undertaken. A total of 124 papers based on adult samples were identified comprising 19 ML studies involving EEG and ECG. A further 21 studies using EEG data, and 84 studies employing ECG meeting all other criteria but not employing ML were included for comparison. Identified studies indicate classical ML methodologies currently dominate EEG and ECG biomarkers research, with derived biomarkers holding clinically relevant diagnostic implications for PTSD. Discussion of the emerging trends, algorithms used and their success is provided, along with areas for future research.

Subtypes of the psychedelic experience have reproducible and predictable effects on depression and anxiety symptoms

BACKGROUND

Subjective experiences seem to play an important role in the enduring effects of psychedelic experiences. Although the importance of the subjective experience on the impact of psychedelics is frequently discussed, a more detailed understanding of the subtypes of psychedelic experiences and their associated impacts on mental health has not been well documented.

METHODS

In the current study, machine learning cluster analysis was used to derive three subtypes of psychedelic experience in a large (n = 985) cross sectional sample.

RESULTS

These subtypes are not only associated with reductions in anxiety and depression symptoms and other markers of psychological wellbeing, but the structure of these subtypes and their subsequent impact on mental health are highly reproducible across multiple psychedelic substances.

LIMITATIONS

Data were obtained via retrospective self-report, which does not allow for definitive conclusions about the direction of causation between baseline characteristics of respondents, qualities of subjective experience, and outcomes.

CONCLUSIONS

The present analysis suggests that psychedelic experiences, in particular those that are associated with enduring improvements in mental health, may be characterized by reproducible and predictable subtypes of the subjective psychedelic effects. These subtypes appear to be significantly different with respect to the baseline demographic characteristics, baseline measures of mental health, and drug type and dose. These findings also suggest that efforts to increase psychedelic associated personal and mystical insight experiences may be key to maximizing beneficial impact of clinical approaches using this treatment in their patients.

Predictive modeling of service discontinuation in transitional age youth with recent behavioral health service use

OBJECTIVE

To develop and test predictive models of discontinuation of behavioral health service use within 12 months in transitional age youth with recent behavioral health service use.

DATA SOURCES

Administrative claims for Medicaid beneficiaries aged 15-26 years in Connecticut.

STUDY DESIGN

We compared the performance of a decision tree, random forest, and gradient boosting machine learning algorithms to logistic regression in predicting service discontinuation within 12 months among beneficiaries using behavioral health services.

DATA EXTRACTION

We identified 33,532 transitional age youth with ≥1 claim for a primary behavioral health diagnosis in 2016 and Medicaid enrollment of ≥11 months in 2016 and ≥11 months in 2017.

PRINCIPAL FINDINGS

Classification accuracy for identifying youth who discontinued behavioral health service use was highest for gradient boosting (80%, AUC = 0.86), decision tree (79%, AUC = 0.84), and random forest (79%, AUC = 0.86), as compared with logistic regression (71%, AUC = 0.71).

CONCLUSIONS

Predictive models based on Medicaid claims can assist in identifying transitional age youth who are at risk of discontinuing from behavioral health care within 12 months, thus allowing for proactive assessment and outreach to promote continuity of care for younger persons who have behavioral health needs.

Deep learning assisted cognitive diagnosis for the D-Riska application

In this article, we expose a system developed that extends the Acquired Brain Injury (ABI) diagnostic application known as D-Riska with an artificial intelligence module that supports the diagnosis of ABI enabling therapists to evaluate patients in an assisted way. The application is in charge of collecting the data of the diagnostic tests of the patients, and due to a multi-class Convolutional Neural Network classifier (CNN), it is capable of making predictions that facilitate the diagnosis and the final score obtained in the test by the patient. To find out the best solution to this problem, different classifiers are used to compare the performance of the proposed model based on various classification metrics. The proposed CNN classifier makes predictions with 93 % of Accuracy, 94 % of Precision, 91 %, of Recall and 92% of F1-Score.

Predicting Final User Satisfaction Using Momentary UX Data and Machine Learning Techniques

User experience (UX) evaluation investigates how people feel about using products or services and is considered an important factor in the design process. However, there is no comprehensive UX evaluation method for time-continuous situations during the use of products or services. Because user experience changes over time, it is difficult to discern the relationship between momentary UX and episodic or cumulative UX, which is related to final user satisfaction. This research aimed to predict final user satisfaction by using momentary UX data and machine learning techniques. The participants were 50 and 25 university students who were asked to evaluate a service (Experiment I) or a product (Experiment II), respectively, during usage by answering a satisfaction survey. Responses were used to draw a customized UX curve. Participants were also asked to complete a final satisfaction questionnaire about the product or service. Momentary UX data and participant satisfaction scores were used to build machine learning models, and the experimental results were compared with those obtained using seven built machine learning models. This study shows that participants’ momentary UX can be understood using a support vector machine (SVM) with a polynomial kernel and that momentary UX can be used to make more accurate predictions about final user satisfaction regarding product and service usage.

Effects of Brain Atlases and Machine Learning Methods on the Discrimination of Schizophrenia Patients: A Multimodal MRI Study

Recently, machine learning techniques have been widely applied in discriminative studies of schizophrenia (SZ) patients with multimodal magnetic resonance imaging (MRI); however, the effects of brain atlases and machine learning methods remain largely unknown. In this study, we collected MRI data for 61 first-episode SZ patients (FESZ), 79 chronic SZ patients (CSZ) and 205 normal controls (NC) and calculated 4 MRI measurements, including regional gray matter volume (GMV), regional homogeneity (ReHo), amplitude of low-frequency fluctuation and degree centrality. We systematically analyzed the performance of two classifications (SZ vs NC; FESZ vs CSZ) based on the combinations of three brain atlases, five classifiers, two cross validation methods and 3 dimensionality reduction algorithms. Our results showed that the groupwise whole-brain atlas with 268 ROIs outperformed the other two brain atlases. In addition, the leave-one-out cross validation was the best cross validation method to select the best hyperparameter set, but the classification performances by different classifiers and dimensionality reduction algorithms were quite similar. Importantly, the contributions of input features to both classifications were higher with the GMV and ReHo features of brain regions in the prefrontal and temporal gyri. Furthermore, an ensemble learning method was performed to establish an integrated model, in which classification performance was improved. Taken together, these findings indicated the effects of these factors in constructing effective classifiers for psychiatric diseases and showed that the integrated model has the potential to improve the clinical diagnosis and treatment evaluation of SZ.

Adaptive interventions for optimizing malaria control: an implementation study protocol for a block-cluster randomized, sequential multiple assignment trial

BACKGROUND

In the past two decades, the massive scale-up of long-lasting insecticidal nets (LLINs) and indoor residual spraying (IRS) has led to significant reductions in malaria mortality and morbidity. Nonetheless, the malaria burden remains high, and a dozen countries in Africa show a trend of increasing malaria incidence over the past several years. This underscores the need to improve the effectiveness of interventions by optimizing first-line intervention tools and integrating newly approved products into control programs. Because transmission settings and vector ecologies vary from place to place, malaria interventions should be adapted and readapted over time in response to evolving malaria risks. An adaptive approach based on local malaria epidemiology and vector ecology may lead to significant reductions in malaria incidence and transmission risk.

METHODS/DESIGN

This study will use a longitudinal block-cluster sequential multiple assignment randomized trial (SMART) design with longitudinal outcome measures for a period of 3 years to develop an adaptive intervention for malaria control in western Kenya, the first adaptive trial for malaria control. The primary outcome is clinical malaria incidence rate. This will be a two-stage trial with 36 clusters for the initial trial. At the beginning of stage 1, all clusters will be randomized with equal probability to either LLIN, piperonyl butoxide-treated LLIN (PBO Nets), or LLIN + IRS by block randomization based on their respective malaria risks. Intervention effectiveness will be evaluated with 12 months of follow-up monitoring. At the end of the 12-month follow-up, clusters will be assessed for "response" versus "non-response" to PBO Nets or LLIN + IRS based on the change in clinical malaria incidence rate and a pre-defined threshold value of cost-effectiveness set by the Ministry of Health. At the beginning of stage 2, if an intervention was effective in stage 1, then the intervention will be continued. Non-responders to stage 1 PBO Net treatment will be randomized equally to either PBO Nets + LSM (larval source management) or an intervention determined by an enhanced reinforcement learning method. Similarly, non-responders to stage 1 LLIN + IRS treatment will be randomized equally to either LLIN + IRS + LSM or PBO Nets + IRS. There will be an 18-month evaluation follow-up period for stage 2 interventions. We will monitor indoor and outdoor vector abundance using light traps. Clinical malaria will be monitored through active case surveillance. Cost-effectiveness of the interventions will be assessed using Q-learning.

DISCUSSION

This novel adaptive intervention strategy will optimize existing malaria vector control tools while allowing for the integration of new control products and approaches in the future to find the most cost-effective malaria control strategies in different settings. Given the urgent global need for optimization of malaria control tools, this study can have far-reaching implications for malaria control and elimination.

TRIAL REGISTRATION

US National Institutes of Health, study ID NCT04182126 . Registered on 26 November 2019.

Can Hyperparameter Tuning Improve the Performance of a Super Learner?: A Case Study

BACKGROUND

Super learning is an ensemble machine learning approach used increasingly as an alternative to classical prediction techniques. When implementing super learning, however, not tuning the hyperparameters of the algorithms in it may adversely affect the performance of the super learner.

METHODS

In this case study, we used data from a Canadian electronic prescribing system to predict when primary care physicians prescribed antidepressants for indications other than depression. The analysis included 73,576 antidepressant prescriptions and 373 candidate predictors. We derived two super learners: one using tuned hyperparameter values for each machine learning algorithm identified through an iterative grid search procedure and the other using the default values. We compared the performance of the tuned super learner to that of the super learner using default values ("untuned") and a carefully constructed logistic regression model from a previous analysis.

RESULTS

The tuned super learner had a scaled Brier score (R) of 0.322 (95% [confidence interval] CI = 0.267, 0.362). In comparison, the untuned super learner had a scaled Brier score of 0.309 (95% CI = 0.256, 0.353), corresponding to an efficiency loss of 4% (relative efficiency 0.96; 95% CI = 0.93, 0.99). The previously-derived logistic regression model had a scaled Brier score of 0.307 (95% CI = 0.245, 0.360), corresponding to an efficiency loss of 5% relative to the tuned super learner (relative efficiency 0.95; 95% CI = 0.88, 1.01).

CONCLUSIONS

In this case study, hyperparameter tuning produced a super learner that performed slightly better than an untuned super learner. Tuning the hyperparameters of individual algorithms in a super learner may help optimize performance.

Likelihood contrasts: a machine learning algorithm for binary classification of longitudinal data

Machine learning methods have gained increased popularity in biomedical research during the recent years. However, very few of them support the analysis of longitudinal data, where several samples are collected from an individual over time. Additionally, most of the available longitudinal machine learning methods assume that the measurements are aligned in time, which is often not the case in real data. Here, we introduce a robust longitudinal machine learning method, named likelihood contrasts (LC), which supports study designs with unaligned time points. Our LC method is a binary classifier, which uses linear mixed models for modelling and log-likelihood for decision making. To demonstrate the benefits of our approach, we compared it with existing methods in four simulated and three real data sets. In each simulated data set, LC was the most accurate method, while the real data sets further supported the robust performance of the method. LC is also computationally efficient and easy to use.

Linear Discriminant Analysis for Prediction of Group Membership: A User-Friendly Primer

In psychology, researchers are often interested in the predictive classification of individuals. Various models exist for such a purpose, but which model is considered a best practice is conditional on attributes of the data. Under certain conditions, linear discriminant analysis (LDA) has been shown to perform better than other predictive methods, such as logistic regression, multinomial logistic regression, random forests, support-vector machines, and the K-nearest neighbor algorithm. The purpose of this Tutorial is to provide researchers who already have a basic level of statistical training with a general overview of LDA and an example of its implementation and interpretation. Decisions that must be made when conducting an LDA (e.g., prior specification, choice of cross-validation procedures) and methods of evaluating case classification (posterior probability, typicality probability) and overall classification (hit rate, Huberty’s I index) are discussed. LDA for prediction is described from a modern Bayesian perspective, as opposed to its original derivation. A step-by-step example of implementing and interpreting LDA results is provided. All analyses were conducted in R, and the script is provided; the data are available online.

Comparative analysis of different characteristics of automatic sleep stages

BACKGROUND AND OBJECTIVE

With the acceleration of social rhythm and the increase of pressure, there are various sleep problems among people. Sleep staging is an important basis for the diagnosis of sleep disorders and other related diseases. The process of automatic staging of sleep is mainly divided into three core steps: data preprocessing, feature extraction, and classification. Accurate analysis of the features of sleep electroencephalogram (EEG) signals is not only helpful to improve the accuracy of sleep staging, but also help people to understand their sleep status.

METHODS

This paper focuses on the analysis of EEG features during sleep staging, and reviews many feature extraction methods and classification methods for sleep staging and sums up these algorithms used in literatures and its staging results. Besides, this paper lists a total of 22 features based on time domain, time-frequency, and nonlinear analysis methods, including kurtosis, skewness, Hjorth parameters, and standard deviations, wavelets energy; sample entropy (SampEn), fuzzy entropy, Tsallis entropy, fractal dimension (FD), complexity. The data set comes from EDF database. Wavelet transform (WT) and support vector machine (SVM) are used to achieve the sleep staging based on single-channel EEG signal. And the characteristic feature data was analyzed by ANOVA.

RESULTS

By comparison, the SampEn, fuzzy entropy, FD and complexity can achieve ideal sleep staging. The highest accuracy of sleep staging is 85.93%. The FD and complexity are simpler than the entropy value, but their accuracies are lower. Furthermore, the distribution of these methods in different sleep period is more significant than others, which is content with the results of sleep staging.

CONCLUSION

In a word, due to the non-stationary and non-linear characteristics of EEG signals, time domain and time-frequency analysis methods all have some limitations. Nonlinear analysis was more effective and practical for the analysis of sleep EEG.

Machine learning in mental health: a scoping review of methods and applications

BACKGROUND

This paper aims to synthesise the literature on machine learning (ML) and big data applications for mental health, highlighting current research and applications in practice.

METHODS

We employed a scoping review methodology to rapidly map the field of ML in mental health. Eight health and information technology research databases were searched for papers covering this domain. Articles were assessed by two reviewers, and data were extracted on the article's mental health application, ML technique, data type, and study results. Articles were then synthesised via narrative review.

RESULTS

Three hundred papers focusing on the application of ML to mental health were identified. Four main application domains emerged in the literature, including: (i) detection and diagnosis; (ii) prognosis, treatment and support; (iii) public health, and; (iv) research and clinical administration. The most common mental health conditions addressed included depression, schizophrenia, and Alzheimer's disease. ML techniques used included support vector machines, decision trees, neural networks, latent Dirichlet allocation, and clustering.

CONCLUSIONS

Overall, the application of ML to mental health has demonstrated a range of benefits across the areas of diagnosis, treatment and support, research, and clinical administration. With the majority of studies identified focusing on the detection and diagnosis of mental health conditions, it is evident that there is significant room for the application of ML to other areas of psychology and mental health. The challenges of using ML techniques are discussed, as well as opportunities to improve and advance the field.

Streamlining Quality Review of Mass Spectrometry Data in the Clinical Laboratory by Use of Machine Learning

Context.—

Turnaround time and productivity of clinical mass spectrometric (MS) testing are hampered by time-consuming manual review of the analytical quality of MS data before release of patient results.

Objective.—

To determine whether a classification model created by using standard machine learning algorithms can verify analytically acceptable MS results and thereby reduce manual review requirements.

Design.—

We obtained retrospective data from gas chromatography–MS analyses of 11-nor-9-carboxy-delta-9-tetrahydrocannabinol (THC-COOH) in 1267 urine samples. The data for each sample had been labeled previously as either analytically unacceptable or acceptable by manual review. The dataset was randomly split into training and test sets (848 and 419 samples, respectively), maintaining equal proportions of acceptable (90%) and unacceptable (10%) results in each set. We used stratified 10-fold cross-validation in assessing the abilities of 6 supervised machine learning algorithms to distinguish unacceptable from acceptable assay results in the training dataset. The classifier with the highest recall was used to build a final model, and its performance was evaluated against the test dataset.

Results.—

In comparison testing of the 6 classifiers, a model based on the Support Vector Machines algorithm yielded the highest recall and acceptable precision. After optimization, this model correctly identified all unacceptable results in the test dataset (100% recall) with a precision of 81%.

Conclusions.—

Automated data review identified all analytically unacceptable assays in the test dataset, while reducing the manual review requirement by about 87%. This automation strategy can focus manual review only on assays likely to be problematic, allowing improved throughput and turnaround time without reducing quality.

Identifying probable post-traumatic stress disorder: applying supervised machine learning to data from a UK military cohort

BACKGROUND

Early identification of probable post-traumatic stress disorder (PTSD) can lead to early intervention and treatment.

AIMS

This study aimed to evaluate supervised machine learning (ML) classifiers for the identification of probable PTSD in those who are serving, or have recently served in the United Kingdom (UK) Armed Forces.

METHODS

Supervised ML classification techniques were applied to a military cohort of 13,690 serving and ex-serving UK Armed Forces personnel to identify probable PTSD based on self-reported service exposures and a range of validated self-report measures. Data were collected between 2004 and 2009.

RESULTS

The predictive performance of supervised ML classifiers to detect cases of probable PTSD were encouraging when compared to a validated measure, demonstrating a capability of supervised ML to detect the cases of probable PTSD. It was possible to identify which variables contributed to the performance, including alcohol misuse, gender and deployment status. A satisfactory sensitivity was obtained across a range of supervised ML classifiers, but sensitivity was low, indicating a potential for false negative diagnoses.

CONCLUSIONS

Detection of probable PTSD based on self-reported measurement data is feasible, may greatly reduce the burden on public health and improve operational efficiencies by enabling early intervention, before manifestation of symptoms.

Replicability of Machine Learning Models in the Social Sciences

Abstract. Machine learning tools are increasingly used in social sciences and policy fields due to their increase in predictive accuracy. However, little research has been done on how well the models of machine learning methods replicate across samples. We compare machine learning methods with regression on the replicability of variable selection, along with predictive accuracy, using an empirical dataset as well as simulated data with additive, interaction, and non-linear squared terms added as predictors. Methods analyzed include support vector machines (SVM), random forests (RF), multivariate adaptive regression splines (MARS), and the regularized regression variants, least absolute shrinkage and selection operator (LASSO), and elastic net. In simulations with additive and linear interactions, machine learning methods performed similarly to regression in replicating predictors; they also performed mostly equal or below regression on measures of predictive accuracy. In simulations with square terms, machine learning methods SVM, RF, and MARS improved predictive accuracy and replicated predictors better than regression. Thus, in simulated datasets, the gap between machine learning methods and regression on predictive measures foreshadowed the gap in variable selection. In replications on the empirical dataset, however, improved prediction by machine learning methods was not accompanied by a visible improvement in replicability in variable selection. This disparity is explained by the overall explanatory power of the models. When predictors have small effects and noise predominates, improved global measures of prediction in a sample by machine learning methods may not lead to the robust selection of predictors; thus, in the presence of weak predictors and noise, regression remains a useful tool for model building and replication.

AIPpred: Sequence-Based Prediction of Anti-inflammatory Peptides Using Random Forest

The use of therapeutic peptides in various inflammatory diseases and autoimmune disorders has received considerable attention; however, the identification of anti-inflammatory peptides (AIPs) through wet-lab experimentation is expensive and often time consuming. Therefore, the development of novel computational methods is needed to identify potential AIP candidates prior to in vitro experimentation. In this study, we proposed a random forest (RF)-based method for predicting AIPs, called AIPpred (AIP predictor in primary amino acid sequences), which was trained with 354 optimal features. First, we systematically studied the contribution of individual composition [amino acid-, dipeptide composition (DPC), amino acid index, chain-transition-distribution, and physicochemical properties] in AIP prediction. Since the performance of the DPC-based model is significantly better than that of other composition-based models, we applied a feature selection protocol on this model and identified the optimal features. AIPpred achieved an area under the curve (AUC) value of 0.801 in a 5-fold cross-validation test, which was ∼2% higher than that of the control RF predictor trained with all DPC composition features, indicating the efficiency of the feature selection protocol. Furthermore, we evaluated the performance of AIPpred on an independent dataset, with results showing that our method outperformed an existing method, as well as 3 different machine learning methods developed in this study, with an AUC value of 0.814. These results indicated that AIPpred will be a useful tool for predicting AIPs and might efficiently assist the development of AIP therapeutics and biomedical research. AIPpred is freely accessible at www.thegleelab.org/AIPpred.

Diagnostic value of structural and diffusion imaging measures in schizophrenia

Objectives

Many studies have attempted to discriminate patients with schizophrenia from healthy controls by machine learning using structural or functional MRI. We included both structural and diffusion MRI (dMRI) and performed random forest (RF) and support vector machine (SVM) in this study.

Methods

We evaluated the performance of classifying schizophrenia using RF method and SVM with 504 features (volume and/or fractional anisotropy and trace) from 184 brain regions. We enrolled 47 patients and 23 age- and sex-matched healthy controls and resampled our data into a balanced dataset using a Synthetic Minority Oversampling Technique method. We randomly permuted the classification of all participants as a patient or healthy control 100 times and ran the RF and SVM with leave one out cross validation for each permutation. We then compared the sensitivity and specificity of the original dataset and the permuted dataset.

Results

Classification using RF with 504 features showed a significantly higher rate of performance compared to classification by chance: sensitivity (87.6% vs. 47.0%) and specificity (95.9 vs. 48.4%) performed by RF, sensitivity (89.5% vs. 48.0%) and specificity (94.5% vs. 47.1%) performed by SVM.

Conclusions

Machine learning using RF and SVM with both volume and diffusion measures can discriminate patients with schizophrenia with a high degree of performance. Further replications are required.

Predictors of Improvements in Mental Health From Mindfulness Meditation in Stressed Older Adults

Context • The benefits of a mindfulness meditation (MM) intervention are most often evidenced by improvements in self-rated stress and mental health. Given the physiological complexity of the psychological stress system, it is likely that some people benefit significantly, whereas others do not. Clinicians and researchers could benefit from further exploration to determine which baseline factors can predict clinically significant improvements from MM. Objectives • The study intended to determine (1) whether the baseline measures for participants who significantly benefitted from MM training were different from the baseline measures of participants who did not, and (2) whether a classification analysis using a decision-tree, machine-learning approach could be useful in predicting which individuals would be most likely to improve. Design • The research team performed a secondary analysis of a previously completed randomized, controlled clinical trial. Setting • The study occurred at the Oregon Health & Science University (Portland, OR, USA) and in participants' homes. Participants • Participants were 134 stressed, generally healthy adults from the metropolitan area of Portland, Oregon, who were 50 to 85 y old. Intervention • Participants were randomly assigned either to a 6-wk MM intervention group or to a waitlist control group, who received the same MM intervention after the waitlist period. Outcome Measures • Outcome measures were assessed at baseline and at 2-mo follow-up intervals. A responder was defined as someone who demonstrated a moderate, clinically significant improvement on the mental health component (MHC) of the short-form health-related quality of life (SF-36) (ie, a change ≥4). The MHC had demonstrated the greatest effect size in the primary analysis of the previously mentioned randomized, controlled clinical trial. Potential predictors were demographic information and baseline measures related to stress and affect. Univariate statistical analyses were performed to compare the values of predictors in the responder and nonresponder groups. In addition, predictors were chosen for a classification analysis using a decision tree approach. Results • Of the 134 original participants, 121 completed the MM intervention. As defined previously, 61 were responders and 60 were nonresponders. Analyses of the baseline measures demonstrated significant differences between the 2 groups in several measures: (1) the positive and negative affect schedule negative subscale (PANAS-neg), (2) the SF-36-MHC, and (3) the SF-36 energy/fatigue, with clinically worse scores being associated with greater likelihood of being a responder. Disappointingly, the decision-tree analyses were unable to achieve a classification rate of better than 65%. Conclusions • The differences in predictor variables between responders and nonresponders to an MM intervention suggested that those with worse mental health at baseline were more likely to improve. Decision-tree analysis was unable to usefully predict who would respond to the intervention.

A review on neuroimaging-based classification studies and associated feature extraction methods for Alzheimer's disease and its prodromal stages

Neuroimaging has made it possible to measure pathological brain changes associated with Alzheimer's disease (AD) in vivo. Over the past decade, these measures have been increasingly integrated into imaging signatures of AD by means of classification frameworks, offering promising tools for individualized diagnosis and prognosis. We reviewed neuroimaging-based studies for AD and mild cognitive impairment classification, selected after online database searches in Google Scholar and PubMed (January, 1985-June, 2016). We categorized these studies based on the following neuroimaging modalities (and sub-categorized based on features extracted as a post-processing step from these modalities): i) structural magnetic resonance imaging [MRI] (tissue density, cortical surface, and hippocampal measurements), ii) functional MRI (functional coherence of different brain regions, and the strength of the functional connectivity), iii) diffusion tensor imaging (patterns along the white matter fibers), iv) fluorodeoxyglucose positron emission tomography (FDG-PET) (metabolic rate of cerebral glucose), and v) amyloid-PET (amyloid burden). The studies reviewed indicate that the classification frameworks formulated on the basis of these features show promise for individualized diagnosis and prediction of clinical progression. Finally, we provided a detailed account of AD classification challenges and addressed some future research directions.

Uncovering Capgras delusion using a large-scale medical records database

BACKGROUND

Capgras delusion is scientifically important but most commonly reported as single case studies. Studies analysing large clinical records databases focus on common disorders but none have investigated rare syndromes.

AIMS

Identify cases of Capgras delusion and associated psychopathology, demographics, cognitive function and neuropathology in light of existing models.

METHOD

Combined computational data extraction and qualitative classification using 250 000 case records from South London and Maudsley Clinical Record Interactive Search (CRIS) database.

RESULTS

We identified 84 individuals and extracted diagnosis-matched comparison groups. Capgras was not 'monothematic' in the majority of cases. Most cases involved misidentified family members or close partners but others were misidentified in 25% of cases, contrary to dual-route face recognition models. Neuroimaging provided no evidence for predominantly right hemisphere damage. Individuals were ethnically diverse with a range of psychosis spectrum diagnoses.

CONCLUSIONS

Capgras is more diverse than current models assume. Identification of rare syndromes complements existing 'big data' approaches in psychiatry.

DECLARATION OF INTERESTS

V.B. is supported by a Wellcome Trust Seed Award in Science (200589/Z/16/Z) and the UCLH NIHR Biomedical Research Centre. S.W. is supported by a Wellcome Trust Strategic Award (WT098455MA). Q.D. has received a grant from King's Health Partners.

COPYRIGHT AND USAGE

© The Royal College of Psychiatrists 2017. This is an open access article distributed under the terms of the Creative Commons Non-Commercial, No Derivatives (CC BY-NC-ND) license.

Evaluation of machine learning algorithms and structural features for optimal MRI-based diagnostic prediction in psychosis

A relatively large number of studies have investigated the power of structural magnetic resonance imaging (sMRI) data to discriminate patients with schizophrenia from healthy controls. However, very few of them have also included patients with bipolar disorder, allowing the clinically relevant discrimination between both psychotic diagnostics. To assess the efficacy of sMRI data for diagnostic prediction in psychosis we objectively evaluated the discriminative power of a wide range of commonly used machine learning algorithms (ridge, lasso, elastic net and L0 norm regularized logistic regressions, a support vector classifier, regularized discriminant analysis, random forests and a Gaussian process classifier) on main sMRI features including grey and white matter voxel-based morphometry (VBM), vertex-based cortical thickness and volume, region of interest volumetric measures and wavelet-based morphometry (WBM) maps. All possible combinations of algorithms and data features were considered in pairwise classifications of matched samples of healthy controls (N = 127), patients with schizophrenia (N = 128) and patients with bipolar disorder (N = 128). Results show that the selection of feature type is important, with grey matter VBM (without data reduction) delivering the best diagnostic prediction rates (averaging over classifiers: schizophrenia vs. healthy 75%, bipolar disorder vs. healthy 63% and schizophrenia vs. bipolar disorder 62%) whereas algorithms usually yielded very similar results. Indeed, those grey matter VBM accuracy rates were not even improved by combining all feature types in a single prediction model. Further multi-class classifications considering the three groups simultaneously made evident a lack of predictive power for the bipolar group, probably due to its intermediate anatomical features, located between those observed in healthy controls and those found in patients with schizophrenia. Finally, we provide MRIPredict (https://www.nitrc.org/projects/mripredict/), a free tool for SPM, FSL and R, to easily carry out voxelwise predictions based on VBM images.

Predicting long-term outcome of Internet-delivered cognitive behavior therapy for social anxiety disorder using fMRI and support vector machine learning

Cognitive behavior therapy (CBT) is an effective treatment for social anxiety disorder (SAD), but many patients do not respond sufficiently and a substantial proportion relapse after treatment has ended. Predicting an individual's long-term clinical response therefore remains an important challenge. This study aimed at assessing neural predictors of long-term treatment outcome in participants with SAD 1 year after completion of Internet-delivered CBT (iCBT). Twenty-six participants diagnosed with SAD underwent iCBT including attention bias modification for a total of 13 weeks. Support vector machines (SVMs), a supervised pattern recognition method allowing predictions at the individual level, were trained to separate long-term treatment responders from nonresponders based on blood oxygen level-dependent (BOLD) responses to self-referential criticism. The Clinical Global Impression-Improvement scale was the main instrument to determine treatment response at the 1-year follow-up. Results showed that the proportion of long-term responders was 52% (12/23). From multivariate BOLD responses in the dorsal anterior cingulate cortex (dACC) together with the amygdala, we were able to predict long-term response rate of iCBT with an accuracy of 92% (confidence interval 95% 73.2-97.6). This activation pattern was, however, not predictive of improvement in the continuous Liebowitz Social Anxiety Scale-Self-report version. Follow-up psychophysiological interaction analyses revealed that lower dACC-amygdala coupling was associated with better long-term treatment response. Thus, BOLD response patterns in the fear-expressing dACC-amygdala regions were highly predictive of long-term treatment outcome of iCBT, and the initial coupling between these regions differentiated long-term responders from nonresponders. The SVM-neuroimaging approach could be of particular clinical value as it allows for accurate prediction of treatment outcome at the level of the individual.