Leakage and the reproducibility crisis in machine-learning-based science

Ensemble machine learning for predicting in-hospital mortality in Asian women with ST-elevation myocardial infarction (STEMI)

The accurate prediction of in-hospital mortality in Asian women after ST-Elevation Myocardial Infarction (STEMI) remains a crucial issue in medical research. Existing models frequently neglect this demographic's particular attributes, resulting in poor treatment outcomes. This study aims to improve the prediction of in-hospital mortality in multi-ethnic Asian women with STEMI by employing both base and ensemble machine learning (ML) models. We centred on the development of demographic-specific models using data from the Malaysian National Cardiovascular Disease Database spanning 2006 to 2016. Through a careful iterative feature selection approach that included feature importance and sequential backward elimination, significant variables such as systolic blood pressure, Killip class, fasting blood glucose, beta-blockers, angiotensin-converting enzyme inhibitors (ACE), and oral hypoglycemic medications were identified. The findings of our study revealed that ML models with selected features outperformed the conventional Thrombolysis in Myocardial Infarction (TIMI) Risk score, with area under the curve (AUC) ranging from 0.60 to 0.93 versus TIMI's AUC of 0.81. Remarkably, our best-performing ensemble ML model was surpassed by the base ML model, support vector machine (SVM) Linear with SVM selected features (AUC: 0.93, CI: 0.89-0.98 versus AUC: 0.91, CI: 0.87-0.96). Furthermore, the women-specific model outperformed a non-gender-specific STEMI model (AUC: 0.92, CI: 0.87-0.97). Our findings demonstrate the value of women-specific ML models over standard approaches, emphasizing the importance of continued testing and validation to improve clinical care for women with STEMI.

Cortical thickness and grey-matter volume anomaly detection in individual MRI scans: Comparison of two methods

Over the past decades, morphometric analysis of brain MRI has contributed substantially to the understanding of healthy brain structure, development and aging as well as to improved characterisation of disease related pathologies. Certified commercial tools based on normative modeling of these metrics are meanwhile available for diagnostic purposes, but they are cost intensive and their clinical evaluation is still in its infancy. Here we have compared the performance of "ScanOMetrics", an open-source research-level tool for detection of statistical anomalies in individual MRI scans, depending on whether it is operated on the output of FreeSurfer or of the deep learning based brain morphometry tool DL + DiReCT. When applied to the public OASIS3 dataset, containing patients with Alzheimer's disease (AD) and healthy controls (HC), cortical thickness anomalies in patient scans were mainly detected in regions that are known as predilection areas of cortical atrophy in AD, regardless of the software used for extraction of the metrics. By contrast, anomaly detections in HCs were up to twenty-fold reduced and spatially unspecific using both DL + DiReCT and FreeSurfer. Progression of the atrophy pattern with clinical dementia rating (CDR) was clearly observable with both methods. DL + DiReCT provided results in less than 25 min, more than 15 times faster than FreeSurfer. This difference in computation time might be relevant when considering application of this or similar methodology as diagnostic decision support for neuroradiologists.

Guided Docking as a Data Generation Approach Facilitates Structure-Based Machine Learning on Kinases

Drug discovery pipelines nowadays rely on machine learning models to explore and evaluate large chemical spaces. While including 3D structural information is considered beneficial, structural models are hindered by the availability of protein-ligand complex structures. Exemplified for kinase drug discovery, we address this issue by generating kinase-ligand complex data using template docking for the kinase compound subset of available ChEMBL assay data. To evaluate the benefit of the created complex data, we use it to train a structure-based E(3)-invariant graph neural network. Our evaluation shows that binding affinities can be predicted with significantly higher precision by models that take synthetic binding poses into account compared to ligand- or drug-target interaction models alone.

Deep learning ensembles for detecting brain metastases in longitudinal multi-modal MRI studies

Metastatic brain cancer is a condition characterized by the migration of cancer cells to the brain from extracranial sites. Notably, metastatic brain tumors surpass primary brain tumors in prevalence by a significant factor, they exhibit an aggressive growth potential and have the capacity to spread across diverse cerebral locations simultaneously. Magnetic resonance imaging (MRI) scans of individuals afflicted with metastatic brain tumors unveil a wide spectrum of characteristics. These lesions vary in size and quantity, spanning from tiny nodules to substantial masses captured within MRI. Patients may present with a limited number of lesions or an extensive burden of hundreds of them. Moreover, longitudinal studies may depict surgical resection cavities, as well as areas of necrosis or edema. Thus, the manual analysis of such MRI scans is difficult, user-dependent and cost-inefficient, and - importantly - it lacks reproducibility. We address these challenges and propose a pipeline for detecting and analyzing brain metastases in longitudinal studies, which benefits from an ensemble of various deep learning architectures originally designed for different downstream tasks (detection and segmentation). The experiments, performed over 275 multi-modal MRI scans of 87 patients acquired in 53 sites, coupled with rigorously validated manual annotations, revealed that our pipeline, built upon open-source tools to ensure its reproducibility, offers high-quality detection, and allows for precisely tracking the disease progression. To objectively quantify the generalizability of models, we introduce a new data stratification approach that accommodates the heterogeneity of the dataset and is used to elaborate training-test splits in a data-robust manner, alongside a new set of quality metrics to objectively assess algorithms. Our system provides a fully automatic and quantitative approach that may support physicians in a laborious process of disease progression tracking and evaluation of treatment efficacy.

Simple Behavioral Analysis (SimBA) as a platform for explainable machine learning in behavioral neuroscience

The study of complex behaviors is often challenging when using manual annotation due to the absence of quantifiable behavioral definitions and the subjective nature of behavioral annotation. Integration of supervised machine learning approaches mitigates some of these issues through the inclusion of accessible and explainable model interpretation. To decrease barriers to access, and with an emphasis on accessible model explainability, we developed the open-source Simple Behavioral Analysis (SimBA) platform for behavioral neuroscientists. SimBA introduces several machine learning interpretability tools, including SHapley Additive exPlanation (SHAP) scores, that aid in creating explainable and transparent behavioral classifiers. Here we show how the addition of explainability metrics allows for quantifiable comparisons of aggressive social behavior across research groups and species, reconceptualizing behavior as a sharable reagent and providing an open-source framework. We provide an open-source, graphical user interface (GUI)-driven, well-documented package to facilitate the movement toward improved automation and sharing of behavioral classification tools across laboratories.

Best practices for machine learning in antibody discovery and development

In the past 40 years, therapeutic antibody discovery and development have advanced considerably, with machine learning (ML) offering a promising way to speed up the process by reducing costs and the number of experiments required. Recent progress in ML-guided antibody design and development (D&D) has been hindered by the diversity of data sets and evaluation methods, which makes it difficult to conduct comparisons and assess utility. Establishing standards and guidelines will be crucial for the wider adoption of ML and the advancement of the field. This perspective critically reviews current practices, highlights common pitfalls and proposes method development and evaluation guidelines for various ML-based techniques in therapeutic antibody D&D. Addressing challenges across the ML process, best practices are recommended for each stage to enhance reproducibility and progress.

Critical Analysis of Data Leakage in WiFi CSI-Based Human Action Recognition Using CNNs

WiFi Channel State Information (CSI)-based human action recognition using convolutional neural networks (CNNs) has emerged as a promising approach for non-intrusive activity monitoring. However, the integrity and reliability of the reported performance metrics are susceptible to data leakage, wherein information from the test set inadvertently influences the training process, leading to inflated accuracy rates. In this paper, we conduct a critical analysis of a notable IEEE Sensors Journal study on WiFi CSI-based human action recognition, uncovering instances of data leakage resulting from the absence of subject-based data partitioning. Empirical investigation corroborates the lack of exclusivity of individuals across dataset partitions, underscoring the importance of rigorous data management practices. Furthermore, we demonstrate that employing data partitioning with respect to humans results in significantly lower precision rates than the reported 99.9% precision, highlighting the exaggerated nature of the original findings. Such inflated results could potentially discourage other researchers and impede progress in the field by fostering a sense of complacency.

Single-channel EOG sleep staging on a heterogeneous cohort of subjects with sleep disorders

Objective.Sleep staging based on full polysomnography is the gold standard in the diagnosis of many sleep disorders. It is however costly, complex, and obtrusive due to the use of multiple electrodes. Automatic sleep staging based on single-channel electro-oculography (EOG) is a promising alternative, requiring fewer electrodes which could be self-applied below the hairline. EOG sleep staging algorithms are however yet to be validated in clinical populations with sleep disorders.Approach.We utilized the SOMNIA dataset, comprising 774 recordings from subjects with various sleep disorders, including insomnia, sleep-disordered breathing, hypersomnolence, circadian rhythm disorders, parasomnias, and movement disorders. The recordings were divided into train (574), validation (100), and test (100) groups. We trained a neural network that integrated transformers within a U-Net backbone. This design facilitated learning of arbitrary-distance temporal relationships within and between the EOG and hypnogram.Main results.For 5-class sleep staging, we achieved median accuracies of 85.0% and 85.2% and Cohen's kappas of 0.781 and 0.796 for left and right EOG, respectively. The performance using the right EOG was significantly better than using the left EOG, possibly because in the recommended AASM setup, this electrode is located closer to the scalp. The proposed model is robust to the presence of a variety of sleep disorders, displaying no significant difference in performance for subjects with a certain sleep disorder compared to those without.Significance.The results show that accurate sleep staging using single-channel EOG can be done reliably for subjects with a variety of sleep disorders.

Diagnostic utility of clinicodemographic, biochemical and metabolite variables to identify viable pregnancies in a symptomatic cohort during early gestation

A significant number of pregnancies are lost in the first trimester and 1-2% are ectopic pregnancies (EPs). Early pregnancy loss in general can cause significant morbidity with bleeding or infection, while EPs are the leading cause of maternal mortality in the first trimester. Symptoms of pregnancy loss and EP are very similar (including pain and bleeding); however, these symptoms are also common in live normally sited pregnancies (LNSP). To date, no biomarkers have been identified to differentiate LNSP from pregnancies that will not progress beyond early gestation (non-viable or EPs), defined together as combined adverse outcomes (CAO). In this study, we present a novel machine learning pipeline to create prediction models that identify a composite biomarker to differentiate LNSP from CAO in symptomatic women. This prospective cohort study included 370 participants. A single blood sample was prospectively collected from participants on first emergency presentation prior to final clinical diagnosis of pregnancy outcome: LNSP, miscarriage, pregnancy of unknown location (PUL) or tubal EP (tEP). Miscarriage, PUL and tEP were grouped together into a CAO group. Human chorionic gonadotrophin β (β-hCG) and progesterone concentrations were measured in plasma. Serum samples were subjected to untargeted metabolomic profiling. The cohort was randomly split into train and validation data sets, with the train data set subjected to variable selection. Nine metabolite signals were identified as key discriminators of LNSP versus CAO. Random forest models were constructed using stable metabolite signals alone, or in combination with plasma hormone concentrations and demographic data. When comparing LNSP with CAO, a model with stable metabolite signals only demonstrated a modest predictive accuracy (0.68), which was comparable to a model of β-hCG and progesterone (0.71). The best model for LNSP prediction comprised stable metabolite signals and hormone concentrations (accuracy = 0.79). In conclusion, serum metabolite levels and biochemical markers from a single blood sample possess modest predictive utility in differentiating LNSP from CAO pregnancies upon first presentation, which is improved by variable selection and combination using machine learning. A diagnostic test to confirm LNSP and thus exclude pregnancies affecting maternal morbidity and potentially life-threatening outcomes would be invaluable in emergency situations.

Advancing Fairness in Cardiac Care: Strategies for Mitigating Bias in Artificial Intelligence Models within Cardiology

In the dynamic field of medical artificial intelligence (AI), cardiology stands out as a key area for its technological advancements and clinical application. This review explores the complex issue of data bias, specifically addressing those encountered during the development and implementation of AI tools in cardiology. We dissect the origins and impacts of these biases, which challenge their reliability and widespread applicability in healthcare. Using a case study, we highlight the complexities involved in addressing these biases from a clinical viewpoint. The goal of this review is to equip researchers and clinicians with the practical knowledge needed to identify, understand, and mitigate these biases, advocating for the creation of AI solutions that are not just technologically sound, but also fair and effective for all patient demographics.

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

Machine learning in cardiac surgery: a narrative review

Background and Objective

Machine learning (ML) is increasingly being utilized to provide data driven solutions to challenges in medicine. Within the field of cardiac surgery, ML methods have been employed as risk stratification tools to predict a variety of operative outcomes. However, the clinical utility of ML in this domain is unclear. The aim of this review is to provide an overview of ML in cardiac surgery, particularly with regards to its utility in predictive analytics and implications for use in clinical decision support.

Methods

We performed a narrative review of relevant articles indexed in PubMed since 2000 using the MeSH terms "Machine Learning", "Supervised Machine Learning", "Deep Learning", or "Artificial Intelligence" and "Cardiovascular Surgery" or "Thoracic Surgery".

Key Content and Findings

ML methods have been widely used to generate pre-operative risk profiles, consistently resulting in the accurate prediction of clinical outcomes in cardiac surgery. However, improvement in predictive performance over traditional risk metrics has proven modest and current applications in the clinical setting remain limited.

Conclusions

Studies utilizing high volume, multidimensional data such as that derived from electronic health record (EHR) data appear to best demonstrate the advantages of ML methods. Models trained on post cardiac surgery intensive care unit data demonstrate excellent predictive performance and may provide greater clinical utility if incorporated as clinical decision support tools. Further development of ML models and their integration into EHR's may result in dynamic clinical decision support strategies capable of informing clinical care and improving outcomes in cardiac surgery.

Predicting non-muscle invasive bladder cancer outcomes using artificial intelligence: a systematic review using APPRAISE-AI

Accurate prediction of recurrence and progression in non-muscle invasive bladder cancer (NMIBC) is essential to inform management and eligibility for clinical trials. Despite substantial interest in developing artificial intelligence (AI) applications in NMIBC, their clinical readiness remains unclear. This systematic review aimed to critically appraise AI studies predicting NMIBC outcomes, and to identify common methodological and reporting pitfalls. MEDLINE, EMBASE, Web of Science, and Scopus were searched from inception to February 5th, 2024 for AI studies predicting NMIBC recurrence or progression. APPRAISE-AI was used to assess methodological and reporting quality of these studies. Performance between AI and non-AI approaches included within these studies were compared. A total of 15 studies (five on recurrence, four on progression, and six on both) were included. All studies were retrospective, with a median follow-up of 71 months (IQR 32-93) and median cohort size of 125 (IQR 93-309). Most studies were low quality, with only one classified as high quality. While AI models generally outperformed non-AI approaches with respect to accuracy, c-index, sensitivity, and specificity, this margin of benefit varied with study quality (median absolute performance difference was 10 for low, 22 for moderate, and 4 for high quality studies). Common pitfalls included dataset limitations, heterogeneous outcome definitions, methodological flaws, suboptimal model evaluation, and reproducibility issues. Recommendations to address these challenges are proposed. These findings emphasise the need for collaborative efforts between urological and AI communities paired with rigorous methodologies to develop higher quality models, enabling AI to reach its potential in enhancing NMIBC care.

Quantification of play behaviour in calves using automated ultra-wideband location data and its association with age, weaning and health status

Play behaviour can act as an indicator of positive animal welfare. Previous attempts to predict play behaviour in farmed calves are limited because of the classification methods used, which lead to overestimation, and the short time periods that calves are observed. The study aimed to automatically classify and quantify play behaviour in farmed calves using location data from ultra-wide band sensors and to investigate factors associated with play behaviour. Location data were collected from 46 calves in three cohorts for a period of 18 weeks. Behavioural observations from video footage were merged with location data to obtain a total of 101.36 h of labelled data. An AdaBoost ensemble learning algorithm was implemented to classify play behaviour. To account for overestimation, generally seen in low-prevalence behaviours, an adjusted count technique was applied to the outputs of the classifier. Two generalized linear mixed models were fitted to investigate factors (e.g. age, health) associated with duration of play and number of play instances per day. Our algorithm identified play behaviour with > 94% accuracy when evaluated on the test set with no animals used for training, and 16% overestimation, which was computed based on the predicted number of samples of play versus the number of samples labelled as play on the test set. The instances and duration of play behaviour per day significantly decreased with age and sickness, whilst play behaviour significantly increased during and after weaning. The instances of play also significantly decreased as mean temperature increased. We suggest that the quantification method that we used could be used to detect and monitor other low prevalence behaviours (e.g. social grooming) from location data, including indicators of positive welfare.

Explainable machine learning for predicting the geographical origin of Chinese Oysters via mineral elements analysis

The traceability of geographic origin is essential for guaranteeing the quality, safety, and protection of oyster brands. However, the current outcomes of traceability lack credibility as they do not adequately explain the model's predictions. Consequently, we conducted a study to evaluate the efficacy of utilizing explainable machine learning combined with mineral elements analysis. The study findings revealed that 18 elements have the ability to determine regional orientation. Simultaneously, individuals should pay closer attention to the potential risks associated with oyster consumption due to the regional differences in essential and toxic elements they contain. Light gradient boosting machine (LightGBM) model exhibited indistinguishable performance, achieving flawless accuracy, precision, recall, F1 score and AUC, with values of 96.77%, 96.43%, 98.53%, 97.32% and 0.998, respectively. The SHapley Additive exPlanations (SHAP) method was used to evaluate the output of the LightGBM model, revealing differences in feature interactions among oysters from different provinces. Specifically, the features Na, Zn, V, Mg, and K were found to have a significant impact on the predictive process of the model. Consistent with existing research, the use of explainable machine learning techniques can provide insights into the complex connections between important product attributes and relevant geographical information.

Deep Learning Models Compared to Experimental Variability for the Prediction of CYP3A4 Time-Dependent Inhibition

Most drugs are mainly metabolized by cytochrome P450 (CYP450), which can lead to drug-drug interactions (DDI). Specifically, time-dependent inhibition (TDI) of CYP3A4 isoenzyme has been associated with clinically relevant DDI. To overcome potential DDI issues, high-throughput in vitro assays were established to assess the TDI of CYP3A4 during the discovery and lead optimization phases. However, in silico machine learning models would enable an earlier and larger-scale assessment of TDI potential liabilities. For CYP inhibition, most modeling efforts have focused on highly imbalanced and small data sets. Moreover, assay variability is rarely considered, which is key to understand the model's quality and suitability for decision-making. In this work, machine learning models were built for the prediction of TDI of CYP3A4, evaluated prospectively, and compared to the variability of the experimental assay. Different modeling strategies were investigated to assess their influence on the model's performance. Through multitask learning, additional data sets were leveraged for model building, coming from public databases, in-house CYP-related assays, or other pharmaceutical companies (federated learning). Apart from the numerical prediction of inactivation rates of CYP3A4 TDI, three-class predictions were carried out, giving a negative (inactivation rate kobs < 0.01 min-1), weak positive (0.01 ≤ kobs ≤ 0.025 min-1), or positive (kobs > 0.025 min-1) output. The final multitask graph neural network model achieved misclassification rates of 8 and 7% for positive and negative TDI, respectively. Importantly, the presented deep learning-based predictions had a similar precision to the reproducibility of in vitro experiments and thus offered great opportunities for drug design, early derisk of DDI potential, and selection of experiments. To facilitate CYP inhibition modeling efforts in the public domain, the developed model was used to annotate ∼16 000 publicly available structures, and a surrogate data set is shared as Supporting Information.

Toward Generalizable Machine Learning Models in Speech, Language, and Hearing Sciences: Estimating Sample Size and Reducing Overfitting

PURPOSE

Many studies using machine learning (ML) in speech, language, and hearing sciences rely upon cross-validations with single data splitting. This study's first purpose is to provide quantitative evidence that would incentivize researchers to instead use the more robust data splitting method of nested k-fold cross-validation. The second purpose is to present methods and MATLAB code to perform power analysis for ML-based analysis during the design of a study.

METHOD

First, the significant impact of different cross-validations on ML outcomes was demonstrated using real-world clinical data. Then, Monte Carlo simulations were used to quantify the interactions among the employed cross-validation method, the discriminative power of features, the dimensionality of the feature space, the dimensionality of the model, and the sample size. Four different cross-validation methods (single holdout, 10-fold, train-validation-test, and nested 10-fold) were compared based on the statistical power and confidence of the resulting ML models. Distributions of the null and alternative hypotheses were used to determine the minimum required sample size for obtaining a statistically significant outcome (5% significance) with 80% power. Statistical confidence of the model was defined as the probability of correct features being selected for inclusion in the final model.

RESULTS

ML models generated based on the single holdout method had very low statistical power and confidence, leading to overestimation of classification accuracy. Conversely, the nested 10-fold cross-validation method resulted in the highest statistical confidence and power while also providing an unbiased estimate of accuracy. The required sample size using the single holdout method could be 50% higher than what would be needed if nested k-fold cross-validation were used. Statistical confidence in the model based on nested k-fold cross-validation was as much as four times higher than the confidence obtained with the single holdout-based model. A computational model, MATLAB code, and lookup tables are provided to assist researchers with estimating the minimum sample size needed during study design.

CONCLUSION

The adoption of nested k-fold cross-validation is critical for unbiased and robust ML studies in the speech, language, and hearing sciences.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.25237045.

Towards a general-purpose foundation model for computational pathology

Quantitative evaluation of tissue images is crucial for computational pathology (CPath) tasks, requiring the objective characterization of histopathological entities from whole-slide images (WSIs). The high resolution of WSIs and the variability of morphological features present significant challenges, complicating the large-scale annotation of data for high-performance applications. To address this challenge, current efforts have proposed the use of pretrained image encoders through transfer learning from natural image datasets or self-supervised learning on publicly available histopathology datasets, but have not been extensively developed and evaluated across diverse tissue types at scale. We introduce UNI, a general-purpose self-supervised model for pathology, pretrained using more than 100 million images from over 100,000 diagnostic H&E-stained WSIs (>77 TB of data) across 20 major tissue types. The model was evaluated on 34 representative CPath tasks of varying diagnostic difficulty. In addition to outperforming previous state-of-the-art models, we demonstrate new modeling capabilities in CPath such as resolution-agnostic tissue classification, slide classification using few-shot class prototypes, and disease subtyping generalization in classifying up to 108 cancer types in the OncoTree classification system. UNI advances unsupervised representation learning at scale in CPath in terms of both pretraining data and downstream evaluation, enabling data-efficient artificial intelligence models that can generalize and transfer to a wide range of diagnostically challenging tasks and clinical workflows in anatomic pathology.

Artificial intelligence and illusions of understanding in scientific research

Scientists are enthusiastically imagining ways in which artificial intelligence (AI) tools might improve research. Why are AI tools so attractive and what are the risks of implementing them across the research pipeline? Here we develop a taxonomy of scientists' visions for AI, observing that their appeal comes from promises to improve productivity and objectivity by overcoming human shortcomings. But proposed AI solutions can also exploit our cognitive limitations, making us vulnerable to illusions of understanding in which we believe we understand more about the world than we actually do. Such illusions obscure the scientific community's ability to see the formation of scientific monocultures, in which some types of methods, questions and viewpoints come to dominate alternative approaches, making science less innovative and more vulnerable to errors. The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less. By analysing the appeal of these tools, we provide a framework for advancing discussions of responsible knowledge production in the age of AI.

Data leakage inflates prediction performance in connectome-based machine learning models

Predictive modeling is a central technique in neuroimaging to identify brain-behavior relationships and test their generalizability to unseen data. However, data leakage undermines the validity of predictive models by breaching the separation between training and test data. Leakage is always an incorrect practice but still pervasive in machine learning. Understanding its effects on neuroimaging predictive models can inform how leakage affects existing literature. Here, we investigate the effects of five forms of leakage-involving feature selection, covariate correction, and dependence between subjects-on functional and structural connectome-based machine learning models across four datasets and three phenotypes. Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects. Furthermore, small datasets exacerbate the effects of leakage. Overall, our results illustrate the variable effects of leakage and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Artificial Intelligence for Drug Discovery: Are We There Yet?

Drug discovery is adapting to novel technologies such as data science, informatics, and artificial intelligence (AI) to accelerate effective treatment development while reducing costs and animal experiments. AI is transforming drug discovery, as indicated by increasing interest from investors, industrial and academic scientists, and legislators. Successful drug discovery requires optimizing properties related to pharmacodynamics, pharmacokinetics, and clinical outcomes. This review discusses the use of AI in the three pillars of drug discovery: diseases, targets, and therapeutic modalities, with a focus on small-molecule drugs. AI technologies, such as generative chemistry, machine learning, and multiproperty optimization, have enabled several compounds to enter clinical trials. The scientific community must carefully vet known information to address the reproducibility crisis. The full potential of AI in drug discovery can only be realized with sufficient ground truth and appropriate human intervention at later pipeline stages.

A critical moment in machine learning in medicine: on reproducible and interpretable learning

Over the past two decades, advances in computational power and data availability combined with increased accessibility to pre-trained models have led to an exponential rise in machine learning (ML) publications. While ML may have the potential to transform healthcare, this sharp increase in ML research output without focus on methodological rigor and standard reporting guidelines has fueled a reproducibility crisis. In addition, the rapidly growing complexity of these models compromises their interpretability, which currently impedes their successful and widespread clinical adoption. In medicine, where failure of such models may have severe implications for patients' health, the high requirements for accuracy, robustness, and interpretability confront ML researchers with a unique set of challenges. In this review, we discuss the semantics of reproducibility and interpretability, as well as related issues and challenges, and outline possible solutions to counteracting the "black box". To foster reproducibility, standard reporting guidelines need to be further developed and data or code sharing encouraged. Editors and reviewers may equally play a critical role by establishing high methodological standards and thus preventing the dissemination of low-quality ML publications. To foster interpretable learning, the use of simpler models more suitable for medical data can inform the clinician how results are generated based on input data. Model-agnostic explanation tools, sensitivity analysis, and hidden layer representations constitute further promising approaches to increase interpretability. Balancing model performance and interpretability are important to ensure clinical applicability. We have now reached a critical moment for ML in medicine, where addressing these issues and implementing appropriate solutions will be vital for the future evolution of the field.

Limited capability of MRI radiomics to predict primary tumor histology of brain metastases in external validation

Background

Growing research demonstrates the ability to predict histology or genetic information of various malignancies using radiomic features extracted from imaging data. This study aimed to investigate MRI-based radiomics in predicting the primary tumor of brain metastases through internal and external validation, using oversampling techniques to address the class imbalance.

Methods

This IRB-approved retrospective multicenter study included brain metastases from lung cancer, melanoma, breast cancer, colorectal cancer, and a combined heterogenous group of other primary entities (5-class classification). Local data were acquired between 2003 and 2021 from 231 patients (545 metastases). External validation was performed with 82 patients (280 metastases) and 258 patients (809 metastases) from the publicly available Stanford BrainMetShare and the University of California San Francisco Brain Metastases Stereotactic Radiosurgery datasets, respectively. Preprocessing included brain extraction, bias correction, coregistration, intensity normalization, and semi-manual binary tumor segmentation. Two-thousand five hundred and twenty-eight radiomic features were extracted from T1w (± contrast), fluid-attenuated inversion recovery (FLAIR), and wavelet transforms for each sequence (8 decompositions). Random forest classifiers were trained with selected features on original and oversampled data (5-fold cross-validation) and evaluated on internal/external holdout test sets using accuracy, precision, recall, F1 score, and area under the receiver-operating characteristic curve (AUC).

Results

Oversampling did not improve the overall unsatisfactory performance on the internal and external test sets. Incorrect data partitioning (oversampling before train/validation/test split) leads to a massive overestimation of model performance.

Conclusions

Radiomics models' capability to predict histologic or genomic data from imaging should be critically assessed; external validation is essential.

Artificial Intelligence and Radiomics for Endometrial Cancer MRI: Exploring the Whats, Whys and Hows

Endometrial cancer (EC) is intricately linked to obesity and diabetes, which are widespread risk factors. Medical imaging, especially magnetic resonance imaging (MRI), plays a major role in EC assessment, particularly for disease staging. However, the diagnostic performance of MRI exhibits variability in the detection of clinically relevant prognostic factors (e.g., deep myometrial invasion and metastatic lymph nodes assessment). To address these challenges and enhance the value of MRI, radiomics and artificial intelligence (AI) algorithms emerge as promising tools with a potential to impact EC risk assessment, treatment planning, and prognosis prediction. These advanced post-processing techniques allow us to quantitatively analyse medical images, providing novel insights into cancer characteristics beyond conventional qualitative image evaluation. However, despite the growing interest and research efforts, the integration of radiomics and AI to EC management is still far from clinical practice and represents a possible perspective rather than an actual reality. This review focuses on the state of radiomics and AI in EC MRI, emphasizing risk stratification and prognostic factor prediction, aiming to illuminate potential advancements and address existing challenges in the field.

The effects of data leakage on connectome-based machine learning models

Predictive modeling has now become a central technique in neuroimaging to identify complex brain-behavior relationships and test their generalizability to unseen data. However, data leakage, which unintentionally breaches the separation between data used to train and test the model, undermines the validity of predictive models. Previous literature suggests that leakage is generally pervasive in machine learning, but few studies have empirically evaluated the effects of leakage in neuroimaging data. Although leakage is always an incorrect practice, understanding the effects of leakage on neuroimaging predictive models provides insight into the extent to which leakage may affect the literature. Here, we investigated the effects of leakage on machine learning models in two common neuroimaging modalities, functional and structural connectomes. Using over 400 different pipelines spanning four large datasets and three phenotypes, we evaluated five forms of leakage fitting into three broad categories: feature selection, covariate correction, and lack of independence between subjects. As expected, leakage via feature selection and repeated subjects drastically inflated prediction performance. Notably, other forms of leakage had only minor effects (e.g., leaky site correction) or even decreased prediction performance (e.g., leaky covariate regression). In some cases, leakage affected not only prediction performance, but also model coefficients, and thus neurobiological interpretations. Finally, we found that predictive models using small datasets were more sensitive to leakage. Overall, our results illustrate the variable effects of leakage on prediction pipelines and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Computational social science is growing up: why puberty consists of embracing measurement validation, theory development, and open science practices

Puberty is a phase in which individuals often test the boundaries of themselves and surrounding others and further define their identity - and thus their uniqueness compared to other individuals. Similarly, as Computational Social Science (CSS) grows up, it must strike a balance between its own practices and those of neighboring disciplines to achieve scientific rigor and refine its identity. However, there are certain areas within CSS that are reluctant to adopt rigorous scientific practices from other fields, which can be observed through an overreliance on passively collected data (e.g., through digital traces, wearables) without questioning the validity of such data. This paper argues that CSS should embrace the potential of combining both passive and active measurement practices to capitalize on the strengths of each approach, including objectivity and psychological quality. Additionally, the paper suggests that CSS would benefit from integrating practices and knowledge from other established disciplines, such as measurement validation, theoretical embedding, and open science practices. Based on this argument, the paper provides ten recommendations for CSS to mature as an interdisciplinary field of research.

Advancing algorithmic drug product development: Recommendations for machine learning approaches in drug formulation

Artificial intelligence is a rapidly expanding area of research, with the disruptive potential to transform traditional approaches in the pharmaceutical industry, from drug discovery and development to clinical practice. Machine learning, a subfield of artificial intelligence, has fundamentally transformed in silico modelling and has the capacity to streamline clinical translation. This paper reviews data-driven modelling methodologies with a focus on drug formulation development. Despite recent advances, there is limited modelling guidance specific to drug product development and a trend towards suboptimal modelling practices, resulting in models that may not give reliable predictions in practice. There is an overwhelming focus on benchtop experimental outcomes obtained for a specific modelling aim, leaving the capabilities of data scraping or the use of combined modelling approaches yet to be fully explored. Moreover, the preference for high accuracy can lead to a reliance on black box methods over interpretable models. This further limits the widespread adoption of machine learning as black boxes yield models that cannot be easily understood for the purposes of enhancing product performance. In this review, recommendations for conducting machine learning research for drug product development to ensure trustworthiness, transparency, and reliability of the models produced are presented. Finally, possible future directions on how research in this area might develop are discussed to aim for models that provide useful and robust guidance to formulators.

Circular and unified analysis in network neuroscience

Genuinely new discovery transcends existing knowledge. Despite this, many analyses in systems neuroscience neglect to test new speculative hypotheses against benchmark empirical facts. Some of these analyses inadvertently use circular reasoning to present existing knowledge as new discovery. Here, I discuss that this problem can confound key results and estimate that it has affected more than three thousand studies in network neuroscience over the last decade. I suggest that future studies can reduce this problem by limiting the use of speculative evidence, integrating existing knowledge into benchmark models, and rigorously testing proposed discoveries against these models. I conclude with a summary of practical challenges and recommendations.

An Open-Source Graphical User Interface-Embedded Automated Electrocardiogram Quality Assessment: A Balanced Class Representation Approach

The rise in cardiovascular diseases necessitates accurate electrocardiogram (ECG) diagnostics, making high-quality ECG recordings essential. Our CNN-LSTM model, embedded in an open-access GUI and trained on balanced datasets collected in clinical settings, excels in automating ECG quality assessment. When tested across three datasets featuring varying ratios of acceptable to unacceptable ECG signals, it achieved an F1 score ranging from 95.87% to 98.40%. Training the model on real noise sources significantly enhances its applicability in real-life scenarios, compared to simulations. Integrated into a user-friendly toolbox, the model offers practical utility in clinical environments. Furthermore, our study underscores the importance of balanced class representation during training and testing phases. We observed a notable F1 score change from 98.09% to 95.87% when the class ratio shifted from 85:15 to 50:50 in the same testing dataset with equal representation. This finding is crucial for future ECG quality assessment research, highlighting the impact of class distribution on the reliability of model training outcomes.

A benchmark dataset for machine learning in ecotoxicology

The use of machine learning for predicting ecotoxicological outcomes is promising, but underutilized. The curation of data with informative features requires both expertise in machine learning as well as a strong biological and ecotoxicological background, which we consider a barrier of entry for this kind of research. Additionally, model performances can only be compared across studies when the same dataset, cleaning, and splittings were used. Therefore, we provide ADORE, an extensive and well-described dataset on acute aquatic toxicity in three relevant taxonomic groups (fish, crustaceans, and algae). The core dataset describes ecotoxicological experiments and is expanded with phylogenetic and species-specific data on the species as well as chemical properties and molecular representations. Apart from challenging other researchers to try and achieve the best model performances across the whole dataset, we propose specific relevant challenges on subsets of the data and include datasets and splittings corresponding to each of these challenge as well as in-depth characterization and discussion of train-test splitting approaches.

A review of PET attenuation correction methods for PET-MR

Despite being thirteen years since the installation of the first PET-MR system, the scanners constitute a very small proportion of the total hybrid PET systems installed. This is in stark contrast to the rapid expansion of the PET-CT scanner, which quickly established its importance in patient diagnosis within a similar timeframe. One of the main hurdles is the development of an accurate, reproducible and easy-to-use method for attenuation correction. Quantitative discrepancies in PET images between the manufacturer-provided MR methods and the more established CT- or transmission-based attenuation correction methods have led the scientific community in a continuous effort to develop a robust and accurate alternative. These can be divided into four broad categories: (i) MR-based, (ii) emission-based, (iii) atlas-based and the (iv) machine learning-based attenuation correction, which is rapidly gaining momentum. The first is based on segmenting the MR images in various tissues and allocating a predefined attenuation coefficient for each tissue. Emission-based attenuation correction methods aim in utilising the PET emission data by simultaneously reconstructing the radioactivity distribution and the attenuation image. Atlas-based attenuation correction methods aim to predict a CT or transmission image given an MR image of a new patient, by using databases containing CT or transmission images from the general population. Finally, in machine learning methods, a model that could predict the required image given the acquired MR or non-attenuation-corrected PET image is developed by exploiting the underlying features of the images. Deep learning methods are the dominant approach in this category. Compared to the more traditional machine learning, which uses structured data for building a model, deep learning makes direct use of the acquired images to identify underlying features. This up-to-date review goes through the literature of attenuation correction approaches in PET-MR after categorising them. The various approaches in each category are described and discussed. After exploring each category separately, a general overview is given of the current status and potential future approaches along with a comparison of the four outlined categories.