Overly optimistic prediction results on imbalanced data: a case study of flaws and benefits when applying over-sampling

Applying oversampling before cross-validation will lead to high bias in radiomics

Class imbalance is often unavoidable for radiomic data collected from clinical routine. It can create problems during classifier training since the majority class could dominate the minority class. Consequently, resampling methods like oversampling or undersampling are applied to the data to class-balance the data. However, the resampling must not be applied upfront to all data because it would lead to data leakage and, therefore, to erroneous results. This study aims to measure the extent of this bias. Five-fold cross-validation with 30 repeats was performed using a set of 15 radiomic datasets to train predictive models. The training involved two scenarios: first, the models were trained correctly by applying the resampling methods during the cross-validation. Second, the models were trained incorrectly by performing the resampling on all the data before cross-validation. The bias was defined empirically as the difference between the best-performing models in both scenarios in terms of area under the receiver operating characteristic curve (AUC), sensitivity, specificity, balanced accuracy, and the Brier score. In addition, a simulation study was performed on a randomly generated dataset for verification. The results demonstrated that incorrectly applying the oversampling methods to all data resulted in a large positive bias (up to 0.34 in AUC, 0.33 in sensitivity, 0.31 in specificity, and 0.37 in balanced accuracy). The bias depended on the data balance, and approximately an increase of 0.10 in the AUC was observed for each increase in imbalance. The models also showed a bias in calibration measured using the Brier score, which differed by up to -0.18 between the correctly and incorrectly trained models. The undersampling methods were not affected significantly by bias. These results emphasize that any resampling method should be applied correctly only to the training data to avoid data leakage and, subsequently, biased model performance and calibration.

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

A soft voting ensemble learning approach for credit card fraud detection

With the advancement of e-commerce and modern technological development, credit cards are widely used for both online and offline purchases, which has increased the number of daily fraudulent transactions. Many organizations and financial institutions worldwide lose billions of dollars annually because of credit card fraud. Due to the global distribution of both legitimate and fraudulent transactions, it is difficult to discern between the two. Furthermore, because only a small proportion of transactions are fraudulent, there is a problem of class imbalance. Hence, an effective fraud-detection methodology is required to sustain the reliability of the payment system. Machine learning has recently emerged as a viable substitute for identifying this type of fraud. However, ML approaches have difficulty identifying fraud with high prediction accuracy, while also decreasing misclassification costs due to the size of the imbalanced data. In this research, a soft voting ensemble learning approach for detecting credit card fraud on imbalanced data is proposed. To do this, the proposed approach is evaluated and compared with numerous sophisticated sampling techniques (i.e., oversampling, undersampling, and hybrid sampling) to overcome the class imbalance problem. We develop several credit card fraud classifiers, including ensemble classifiers, with and without sampling techniques. According to the experimental results, the proposed soft-voting approach outperforms individual classifiers. With a false negative rate (FNR) of 0.0306, it achieves a precision of 0.9870, recall of 0.9694, f1-score of 0.8764, and AUROC of 0.9936.

Multimodal Data Integration Advances Longitudinal Prediction of the Naturalistic Course of Depression and Reveals a Multimodal Signature of Remission During 2-Year Follow-up

BACKGROUND

The ability to predict the disease course of individuals with major depressive disorder (MDD) is essential for optimal treatment planning. Here, we used a data-driven machine learning approach to assess the predictive value of different sets of biological data (whole-blood proteomics, lipid metabolomics, transcriptomics, genetics), both separately and added to clinical baseline variables, for the longitudinal prediction of 2-year remission status in MDD at the individual-subject level.

METHODS

Prediction models were trained and cross-validated in a sample of 643 patients with current MDD (2-year remission n = 325) and subsequently tested for performance in 161 individuals with MDD (2-year remission n = 82).

RESULTS

Proteomics data showed the best unimodal data predictions (area under the receiver operating characteristic curve = 0.68). Adding proteomic to clinical data at baseline significantly improved 2-year MDD remission predictions (area under the receiver operating characteristic curve = 0.63 vs. 0.78, p = .013), while the addition of other omics data to clinical data did not yield significantly improved model performance. Feature importance and enrichment analysis revealed that proteomic analytes were involved in inflammatory response and lipid metabolism, with fibrinogen levels showing the highest variable importance, followed by symptom severity. Machine learning models outperformed psychiatrists' ability to predict 2-year remission status (balanced accuracy = 71% vs. 55%).

CONCLUSIONS

This study showed the added predictive value of combining proteomic data, but not other omics data, with clinical data for the prediction of 2-year remission status in MDD. Our results reveal a novel multimodal signature of 2-year MDD remission status that shows clinical potential for individual MDD disease course predictions from baseline measurements.

Development and validation of a digital biomarker predicting acute kidney injury following cardiac surgery on an hourly basis

Objectives

To develop and validate a digital biomarker for predicting the onset of acute kidney injury (AKI) on an hourly basis up to 24 hours in advance in the intensive care unit after cardiac surgery.

Methods

The study analyzed data from 6056 adult patients undergoing coronary artery bypass graft and/or valve surgery between April 1, 2012, and December 31, 2018 (development phase, training, and testing) and 3572 patients between January 1, 2019, and June 30, 2022 (validation phase). The study used 2 dynamic predictive modeling approaches, namely logistic regression and bootstrap aggregated regression trees machine (BARTm), to predict AKI. The mean area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and positive and negative predictive values across all lead times before the occurrence of AKI were reported. The clinical practicality was assessed using calibration.

Results

Of all included patients, 8.45% and 16.66% had AKI in the development and validation phases, respectively. When applied to testing data, AKI was predicted with the mean AUC of 0.850 and 0.802 by BARTm and logistic regression, respectively. When applied to validation data, BARTm and LR resulted in a mean AUC of 0.844 and 0.786, respectively.

Conclusions

This study demonstrated the successful prediction of AKI on an hourly basis up to 24 hours in advance. The digital biomarkers developed and validated in this study have the potential to assist clinicians in optimizing treatment and implementing preventive strategies for patients at risk of developing AKI after cardiac surgery in the intensive care unit.

Multi-channel electrohysterography enabled uterine contraction characterization and its effect in delivery assessment

Uterine contractions are routinely monitored by tocodynamometer (TOCO) at late stage of pregnancy to predict the onset of labor. However, TOCO reveals no information on the synchrony and coherence of contractions, which are important contributors to a successful delivery. The electrohysterography (EHG) is a recording of the electrical activities that trigger the local muscles to contract. The spatial-temporal information embedded in multiple channel EHG signals make them ideal for characterizing the synchrony and coherence of uterine contraction. To proceed, contractile time-windows are identified from TOCO signals and are then used to segment out the simultaneously recorded EHG signals of different channels. We construct sample entropy SamEn and Concordance Correlation based feature ψ from these EHG segments to quantify the synchrony and coherence of contraction. To test the effectiveness of the proposed method, 122 EHG recordings in the Icelandic EHG database were divided into two groups according to the time difference between the gestational ages at recording and at delivery (TTD). Both SamEn and ψ show clear difference in the two groups (p<10-5) even when measurements were made 120 h before delivery. Receiver operating characteristic curve analysis of these two features gave AUC values of 0.834 and 0.726 for discriminating imminent labor defined with TTD ≤ 24 h. The SamEn was significantly smaller in women (0.1433) of imminent labor group than in women (0.3774) of the pregnancy group. Using an optimal cutoff value of SamEn to identify imminent labor gives sensitivity, specificity, and accuracy as high as 0.909, 0.712 and 0.743, respectively. These results demonstrate superiority in comparing to the existing SOTA methods. This study is the first research work focusing on characterizing the synchrony property of contractions from the electrohysterography signals. Despite the very limited dataset used in the validation process, the promising results open a new direction to the use of electrohysterography in obstetrics.

An open dataset with electrohysterogram records of pregnancies ending in induced and cesarean section delivery

The existing non-invasive automated preterm birth prediction methods rely on the use of uterine electrohysterogram (EHG) records coming from spontaneous preterm and term deliveries, and are indifferent to term induced and cesarean section deliveries. In order to enhance current publicly available pool of term EHG records, we developed a new EHG dataset, Induced Cesarean EHG DataSet (ICEHG DS), containing 126 30-minute EHG records, recorded early (23rd week), and/or later (31st week) during pregnancy, of those pregnancies that were expected to end in spontaneous term delivery, but ended in induced or cesarean section delivery. The records were collected at the University Medical Center Ljubljana, Ljubljana, Slovenia. The dataset includes 38 and 43, early and later, induced; 11 and 8, early and later, cesarean; and 13 and 13, early and later, induced and cesarean EHG records. This dataset enables better understanding of the underlying physiological mechanisms involved during pregnancies ending in induced and cesarean deliveries, and provides a robust and more realistic assessment of the performance of automated preterm birth prediction methods.

Leakage and the reproducibility crisis in machine-learning-based science

Machine-learning (ML) methods have gained prominence in the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. We systematically investigate reproducibility issues in ML-based science. Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions. Based on our survey, we introduce a detailed taxonomy of eight types of leakage, ranging from textbook errors to open research problems. We propose that researchers test for each type of leakage by filling out model info sheets, which we introduce. Finally, we conduct a reproducibility study of civil war prediction, where complex ML models are believed to vastly outperform traditional statistical models such as logistic regression (LR). When the errors are corrected, complex ML models do not perform substantively better than decades-old LR models.

3D CT-Inclusive Deep-Learning Model to Predict Mortality, ICU Admittance, and Intubation in COVID-19 Patients

Chest CT is a useful initial exam in patients with coronavirus disease 2019 (COVID-19) for assessing lung damage. AI-powered predictive models could be useful to better allocate resources in the midst of the pandemic. Our aim was to build a deep-learning (DL) model for COVID-19 outcome prediction inclusive of 3D chest CT images acquired at hospital admission. This retrospective multicentric study included 1051 patients (mean age 69, SD = 15) who presented to the emergency department of three different institutions between 20th March 2020 and 20th January 2021 with COVID-19 confirmed by real-time reverse transcriptase polymerase chain reaction (RT-PCR). Chest CT at hospital admission were evaluated by a 3D residual neural network algorithm. Training, internal validation, and external validation groups included 608, 153, and 290 patients, respectively. Images, clinical, and laboratory data were fed into different customizations of a dense neural network to choose the best performing architecture for the prediction of mortality, intubation, and intensive care unit (ICU) admission. The AI model tested on CT and clinical features displayed accuracy, sensitivity, specificity, and ROC-AUC, respectively, of 91.7%, 90.5%, 92.4%, and 95% for the prediction of patient's mortality; 91.3%, 91.5%, 89.8%, and 95% for intubation; and 89.6%, 90.2%, 86.5%, and 94% for ICU admission (internal validation) in the testing cohort. The performance was lower in the validation cohort for mortality (71.7%, 55.6%, 74.8%, 72%), intubation (72.6%, 74.7%, 45.7%, 64%), and ICU admission (74.7%, 77%, 46%, 70%) prediction. The addition of the available laboratory data led to an increase in sensitivity for patient's mortality (66%) and specificity for intubation and ICU admission (50%, 52%, respectively), while the other metrics maintained similar performance results. We present a deep-learning model to predict mortality, ICU admittance, and intubation in COVID-19 patients. KEY POINTS: • 3D CT-based deep learning model predicted the internal validation set with high accuracy, sensibility and specificity (> 90%) mortality, ICU admittance, and intubation in COVID-19 patients. • The model slightly increased prediction results when laboratory data were added to the analysis, despite data imbalance. However, the model accuracy dropped when CT images were not considered in the analysis, implying an important role of CT in predicting outcomes.

End-to-end learning with interpretation on electrohysterography data to predict preterm birth

Prediction of preterm birth is a difficult task for clinicians. By examining an electrohysterogram, electrical activity of the uterus that can lead to preterm birth can be detected. Since signals associated with uterine activity are difficult to interpret for clinicians without a background in signal processing, machine learning may be a viable solution. We are the first to employ Deep Learning models, a long-short term memory and temporal convolutional network model, on electrohysterography data using the Term-Preterm Electrohysterogram database. We show that end-to-end learning achieves an AUC score of 0.58, which is comparable to machine learning models that use handcrafted features. Moreover, we evaluate the effect of adding clinical data to the model and conclude that adding the available clinical data to electrohysterography data does not result in a gain in performance. Also, we propose an interpretability framework for time series classification that is well-suited to use in case of limited data, as opposed to existing methods that require large amounts of data. Clinicians with extensive work experience as gynaecologist used our framework to provide insights on how to link our results to clinical practice and stress that in order to decrease the number of false positives, a dataset with patients at high risk of preterm birth should be collected. All code is made publicly available.

Evaluation of a decided sample size in machine learning applications

BACKGROUND

An appropriate sample size is essential for obtaining a precise and reliable outcome of a study. In machine learning (ML), studies with inadequate samples suffer from overfitting of data and have a lower probability of producing true effects, while the increment in sample size increases the accuracy of prediction but may not cause a significant change after a certain sample size. Existing statistical approaches using standardized mean difference, effect size, and statistical power for determining sample size are potentially biased due to miscalculations or lack of experimental details. This study aims to design criteria for evaluating sample size in ML studies. We examined the average and grand effect sizes and the performance of five ML methods using simulated datasets and three real datasets to derive the criteria for sample size. We systematically increase the sample size, starting from 16, by randomly sampling and examine the impact of sample size on classifiers' performance and both effect sizes. Tenfold cross-validation was used to quantify the accuracy.

RESULTS

The results demonstrate that the effect sizes and the classification accuracies increase while the variances in effect sizes shrink with the increment of samples when the datasets have a good discriminative power between two classes. By contrast, indeterminate datasets had poor effect sizes and classification accuracies, which did not improve by increasing sample size in both simulated and real datasets. A good dataset exhibited a significant difference in average and grand effect sizes. We derived two criteria based on the above findings to assess a decided sample size by combining the effect size and the ML accuracy. The sample size is considered suitable when it has appropriate effect sizes (≥ 0.5) and ML accuracy (≥ 80%). After an appropriate sample size, the increment in samples will not benefit as it will not significantly change the effect size and accuracy, thereby resulting in a good cost-benefit ratio.

CONCLUSION

We believe that these practical criteria can be used as a reference for both the authors and editors to evaluate whether the selected sample size is adequate for a study.

Predicting preterm births from electrohysterogram recordings via deep learning

About one in ten babies is born preterm, i.e., before completing 37 weeks of gestation, which can result in permanent neurologic deficit and is a leading cause of child mortality. Although imminent preterm labor can be detected, predicting preterm births more than one week in advance remains elusive. Here, we develop a deep learning method to predict preterm births directly from electrohysterogram (EHG) measurements of pregnant mothers recorded at around 31 weeks of gestation. We developed a prediction model, which includes a recurrent neural network, to predict preterm births using short-time Fourier transforms of EHG recordings and clinical information from two public datasets. We predicted preterm births with an area under the receiver-operating characteristic curve (AUC) of 0.78 (95% confidence interval: 0.76-0.80). Moreover, we found that the spectral patterns of the measurements were more predictive than the temporal patterns, suggesting that preterm births can be predicted from short EHG recordings in an automated process. We show that preterm births can be predicted for pregnant mothers around their 31st week of gestation, prompting beneficial treatments to reduce the incidence of preterm births and improve their outcomes.

Machine Learning Models for Data-Driven Prediction of Diabetes by Lifestyle Type

The prevalence of diabetes has been increasing in recent years, and previous research has found that machine-learning models are good diabetes prediction tools. The purpose of this study was to compare the efficacy of five different machine-learning models for diabetes prediction using lifestyle data from the National Health and Nutrition Examination Survey (NHANES) database. The 1999-2020 NHANES database yielded data on 17,833 individuals data based on demographic characteristics and lifestyle-related variables. To screen training data for machine models, the Akaike Information Criterion (AIC) forward propagation algorithm was utilized. For predicting diabetes, five machine-learning models (CATBoost, XGBoost, Random Forest (RF), Logistic Regression (LR), and Support Vector Machine (SVM)) were developed. Model performance was evaluated using accuracy, sensitivity, specificity, precision, F1 score, and receiver operating characteristic (ROC) curve. Among the five machine-learning models, the dietary intake levels of energy, carbohydrate, and fat, contributed the most to the prediction of diabetes patients. In terms of model performance, CATBoost ranks higher than RF, LG, XGBoost, and SVM. The best-performing machine-learning model among the five is CATBoost, which achieves an accuracy of 82.1% and an AUC of 0.83. Machine-learning models based on NHANES data can assist medical institutions in identifying diabetes patients.

Artificial intelligence for precision medicine in autoimmune liver disease

Autoimmune liver diseases (AiLDs) are rare autoimmune conditions of the liver and the biliary tree with unknown etiology and limited treatment options. AiLDs are inherently characterized by a high degree of complexity, which poses great challenges in understanding their etiopathogenesis, developing novel biomarkers and risk-stratification tools, and, eventually, generating new drugs. Artificial intelligence (AI) is considered one of the best candidates to support researchers and clinicians in making sense of biological complexity. In this review, we offer a primer on AI and machine learning for clinicians, and discuss recent available literature on its applications in medicine and more specifically how it can help to tackle major unmet needs in AiLDs.

Predictors of improvement in quality of life at 12-month follow-up in patients undergoing anterior endoscopic skull base surgery

Background

Patients with pituitary lesions experience decrements in quality of life (QoL) and treatment aims to arrest or improve QoL decline.

Objective

To detect associations with QoL in trans-nasal endoscopic skull base surgery patients and train supervised learning classifiers to predict QoL improvement at 12 months.

Methods

A supervised learning analysis of a prospective multi-institutional dataset (451 patients) was conducted. QoL was measured using the anterior skull base surgery questionnaire (ASBS). Factors associated with QoL at baseline and at 12-month follow-up were identified using multivariate logistic regression. Multiple supervised learning models were trained to predict postoperative QoL improvement with five-fold cross-validation.

Results

ASBS at 12-month follow-up was significantly higher (132.19,SD = 24.87) than preoperative ASBS (121.87,SD = 25.72,p<0.05). High preoperative scores were significantly associated with institution, diabetes and lesions at the planum sphenoidale / tuberculum sella site. Patients with diabetes were five times less likely to report high preoperative QoL. Low preoperative QoL was significantly associated with female gender, a vision-related presentation, diabetes, secreting adenoma and the cavernous sinus site. Top quartile change in postoperative QoL at 12-month follow-up was negatively associated with baseline hypercholesterolemia, acromegaly and intraoperative CSF leak. Positive associations were detected for lesions at the sphenoid sinus site and deficient preoperative endocrine function. AdaBoost, logistic regression and neural network classifiers yielded the strongest predictive performance.

Conclusion

It was possible to predict postoperative positive change in QoL at 12-month follow-up using perioperative data. Further development and implementation of these models may facilitate improvements in informed consent, treatment decision-making and patient QoL.

Combination of Feature Selection and Resampling Methods to Predict Preterm Birth Based on Electrohysterographic Signals from Imbalance Data

Due to its high sensitivity, electrohysterography (EHG) has emerged as an alternative technique for predicting preterm labor. The main obstacle in designing preterm labor prediction models is the inherent preterm/term imbalance ratio, which can give rise to relatively low performance. Numerous studies obtained promising preterm labor prediction results using the synthetic minority oversampling technique. However, these studies generally overestimate mathematical models’ real generalization capacity by generating synthetic data before splitting the dataset, leaking information between the training and testing partitions and thus reducing the complexity of the classification task. In this work, we analyzed the effect of combining feature selection and resampling methods to overcome the class imbalance problem for predicting preterm labor by EHG. We assessed undersampling, oversampling, and hybrid methods applied to the training and validation dataset during feature selection by genetic algorithm, and analyzed the resampling effect on training data after obtaining the optimized feature subset. The best strategy consisted of undersampling the majority class of the validation dataset to 1:1 during feature selection, without subsequent resampling of the training data, achieving an AUC of 94.5 ± 4.6%, average precision of 84.5 ± 11.7%, maximum F1-score of 79.6 ± 13.8%, and recall of 89.8 ± 12.1%. Our results outperformed the techniques currently used in clinical practice, suggesting the EHG could be used to predict preterm labor in clinics.

Comparing Sampling Strategies for Tackling Imbalanced Data in Human Activity Recognition

Human activity recognition (HAR) using wearable sensors is an increasingly active research topic in machine learning, aided in part by the ready availability of detailed motion capture data from smartphones, fitness trackers, and smartwatches. The goal of HAR is to use such devices to assist users in their daily lives in application areas such as healthcare, physical therapy, and fitness. One of the main challenges for HAR, particularly when using supervised learning methods, is obtaining balanced data for algorithm optimisation and testing. As people perform some activities more than others (e.g., walk more than run), HAR datasets are typically imbalanced. The lack of dataset representation from minority classes hinders the ability of HAR classifiers to sufficiently capture new instances of those activities. We introduce three novel hybrid sampling strategies to generate more diverse synthetic samples to overcome the class imbalance problem. The first strategy, which we call the distance-based method (DBM), combines Synthetic Minority Oversampling Techniques (SMOTE) with Random_SMOTE, both of which are built around the k-nearest neighbors (KNN). The second technique, referred to as the noise detection-based method (NDBM), combines SMOTE Tomek links (SMOTE_Tomeklinks) and the modified synthetic minority oversampling technique (MSMOTE). The third approach, which we call the cluster-based method (CBM), combines Cluster-Based Synthetic Oversampling (CBSO) and Proximity Weighted Synthetic Oversampling Technique (ProWSyn). We compare the performance of the proposed hybrid methods to the individual constituent methods and baseline using accelerometer data from three commonly used benchmark datasets. We show that DBM, NDBM, and CBM reduce the impact of class imbalance and enhance F1 scores by a range of 9-20 percentage point compared to their constituent sampling methods. CBM performs significantly better than the others under a Friedman test, however, DBM has lower computational requirements.

Network theory based EHG signal analysis and its application in preterm prediction

OBJECTIVE

Preterm birth is the leading cause of neonatal morbidity and mortality. Early identification of high-risk patients followed by medical interventions is essential to the prevention of preterm birth. Based on the relationship between uterine contraction and the fundamental electrical activities of muscles, we extracted effective features from EHG signals recorded from pregnant women, and use them to train classifiers with the purpose of providing high precision in classifying term and preterm pregnancies.

METHODS

To characterize changes from irregularity to coherence of the uterine activity during the whole pregnancy, network representations of the original electrohysterogram (EHG) signals are established by applying the Horizontal Visibility Graph (HVG) algorithm, from which we extract network degree density and distribution, clustering coefficient and assortativity coefficient. Concerns on the interferences of different noise sources embedded in the EHG signal, we apply Short-Time Fourier Transform (STFT) to expand the original signal in the time-frequency domain. This allows a network representation and the extraction of related features on each frequency component. Feature selection algorithms are then used to filter out unrelated frequency components. We further apply the proposed feature extraction method to EHG signals available in the Term-Preterm EHG database (TPEHG), and use them to train classifiers. We adopt the Partition-Synthesis scheme which splits the original imbalanced dataset into two sets and synthesizes artificial samples separately within each subset to solve the problem of dataset imbalance.

RESULTS

The optimally selected network-based features, not only contribute to the identification of the essential frequency components of uterine activities related to preterm birth, but also to improved performance in classifying term/preterm pregnancies, i.e., the SVM (Support Vector Machine) classifier trained with the available samples in the TPEHG gives sensitivity, specificity, overall accuracy, and auc values as high as 0.89, 0.93, 0.91, and 0.97, respectively.

A new baseline for retinal vessel segmentation: Numerical identification and correction of methodological inconsistencies affecting 100+ papers

In the last 15 years, the segmentation of vessels in retinal images has become an intensively researched problem in medical imaging, with hundreds of algorithms published. One of the de facto benchmarking data sets of vessel segmentation techniques is the DRIVE data set. Since DRIVE contains a predefined split of training and test images, the published performance results of the various segmentation techniques should provide a reliable ranking of the algorithms. Including more than 100 papers in the study, we performed a detailed numerical analysis of the coherence of the published performance scores. We found inconsistencies in the reported scores related to the use of the field of view (FoV), which has a significant impact on the performance scores. We attempted to eliminate the biases using numerical techniques to provide a more realistic picture of the state of the art. Based on the results, we have formulated several findings, most notably: despite the well-defined test set of DRIVE, most rankings in published papers are based on non-comparable figures; in contrast to the near-perfect accuracy scores reported in the literature, the highest accuracy score achieved to date is 0.9582 in the FoV region, which is 1% higher than that of human annotators. The methods we have developed for identifying and eliminating the evaluation biases can be easily applied to other domains where similar problems may arise.

Alvarez waves in pregnancy: a comprehensive review

Alvarez waves are local rhythmic contractions of the myometrium with high frequency and low intensity. They can be detected using internal or external tocography and electrohysterography. Some researchers correlate these small contractions with the initiation of labor, since they have been described as a pattern representing the uterine response to prostaglandin production. Other authors either do not validate a causality relation between Alvarez waves and labor or suggest that they have low predictive value for preterm labor. Alvarez waves' research has become a multidisciplinary subject with inputs ranging from medical science, biomedical engineering, and related areas. A comprehensive review is herein conducted to summarize the state of the art regarding Alvarez waves and their role in the initiation of labor, namely in preterm birth. The results show that a large number of studies have analyzed and characterized Alvarez waves without necessarily digging into their relationship with labor. Publications were categorized in three groups: (A) reports about morphology and characterization of Alvarez waves; (B) publications reporting a positive causality relation between Alvarez waves and labor; and (C) publications reporting an absence of causality regarding the previous hypothesis. Studies in group B outnumbered those in group C. A critical analysis is presented.

Class-Imbalanced Voice Pathology Detection and Classification Using Fuzzy Cluster Oversampling Method

The Massachusetts Eye and Ear Infirmary (MEEI) database is an international-standard training database for voice pathology detection (VPD) systems. However, there is a class-imbalanced distribution in normal and pathological voice samples and different types of pathological voice samples in the MEEI database. This study aimed to develop a VPD system that uses the fuzzy clustering synthetic minority oversampling technique algorithm (FC-SMOTE) to automatically detect and classify four types of pathological voices in a multi-class imbalanced database. The proposed FC-SMOTE algorithm processes the initial class-imbalanced dataset. A set of machine learning models was evaluated and validated using the resulting class-balanced dataset as an input. The effectiveness of the VPD system with FC-SMOTE was further verified by an external validation set and another pathological voice database (Saarbruecken Voice Database (SVD)). The experimental results show that, in the multi-classification of pathological voice for the class-imbalanced dataset, the method we propose can significantly improve the diagnostic accuracy. Meanwhile, FC-SMOTE outperforms the traditional imbalanced data oversampling algorithms, and it is preferred for imbalanced voice diagnosis in practical applications.

Acoustic and language analysis of speech for suicidal ideation among US veterans

BACKGROUND

Screening for suicidal ideation in high-risk groups such as U.S. veterans is crucial for early detection and suicide prevention. Currently, screening is based on clinical interviews or self-report measures. Both approaches rely on subjects to disclose their suicidal thoughts. Innovative approaches are necessary to develop objective and clinically applicable assessments. Speech has been investigated as an objective marker to understand various mental states including suicidal ideation. In this work, we developed a machine learning and natural language processing classifier based on speech markers to screen for suicidal ideation in US veterans.

METHODOLOGY

Veterans submitted 588 narrative audio recordings via a mobile app in a real-life setting. In addition, participants completed self-report psychiatric scales and questionnaires. Recordings were analyzed to extract voice characteristics including prosodic, phonation, and glottal. The audios were also transcribed to extract textual features for linguistic analysis. We evaluated the acoustic and linguistic features using both statistical significance and ensemble feature selection. We also examined the performance of different machine learning algorithms on multiple combinations of features to classify suicidal and non-suicidal audios.

RESULTS

A combined set of 15 acoustic and linguistic features of speech were identified by the ensemble feature selection. Random Forest classifier, using the selected set of features, correctly identified suicidal ideation in veterans with 86% sensitivity, 70% specificity, and an area under the receiver operating characteristic curve (AUC) of 80%.

CONCLUSIONS

Speech analysis of audios collected from veterans in everyday life settings using smartphones offers a promising approach for suicidal ideation detection. A machine learning classifier may eventually help clinicians identify and monitor high-risk veterans.