Interpretability in the medical field: A systematic mapping and review study

Interpretable machine learning to evaluate relationships between DAO/DAOA (pLG72) protein data and features in clinical assessments, functional outcome, and cognitive function in schizophrenia patients

Machine learning has been proposed to utilize D-amino acid oxidase (DAO) and DAO activator (DAOA [or pLG72]) protein levels to ascertain disease status in schizophrenia. However, it remains unclear whether machine learning can effectively evaluate clinical features in relation to DAO and DAOA in schizophrenia patients. We employed an interpretable machine learning (IML) framework including linear regression, least absolute shrinkage and selection operator (Lasso) models, and generalized additive models (GAMs) to analyze DAO/DAOA levels using 380 Taiwanese schizophrenia patients. Additionally, we incorporated 27 parameters encompassing demographic variables, clinical assessments, functional outcomes, and cognitive function as features. The IML framework facilitated linear and non-linear relationships between features and DAO/DAOA. DAO levels demonstrated significant associations with the 17-item Hamilton Depression Rating Scale (HAMD17) based on linear regression. The Lasso model identified four features-HAMD17, age, working memory, and overall cognitive function (OCF)-and highlighted HAMD17 as the most significant feature, using DAO from chronically stable patients. Utilizing DAOA from acutely exacerbated patients, the Lasso model also identified four features-OCF, Scale for the Assessment of Negative Symptoms 20-item, quality of life scale (QLS), and category fluency-and emphasized OCF as the most significant feature. Furthermore, GAMs revealed a non-linear relationship between category fluency and DAO in chronically stable patients, as well as between QLS and DAOA in acutely exacerbated patients. The study suggests that an IML framework holds promise for assessing linear and non-linear relationships between DAO/DAOA and various features in clinical assessments, functional outcomes, and cognitive function in patients with schizophrenia.

Construction of a prognostic prediction model for colorectal cancer based on 5-year clinical follow-up data

Colorectal cancer (CRC) is a prevalent malignant tumor that presents significant challenges to both public health and healthcare systems. The aim of this study was to develop a machine learning model based on five years of clinical follow-up data from CRC patients to accurately identify individuals at risk of poor prognosis. This study included 411 CRC patients who underwent surgery at Yixing Hospital and completed the follow-up process. A modeling dataset containing 73 characteristic variables was established by collecting demographic information, clinical blood test indicators, histopathological results, and additional treatment-related information. Decision tree, random forest, support vector machine, and extreme gradient boosting (XGBoost) models were selected for modeling based on the features identified through recursive feature elimination (RFE). The Cox proportional hazards model was used as the baseline for model comparison. During the model training process, hyperparameters were optimized using a grid search method. The model performance was comprehensively assessed using multiple metrics, including accuracy, F1 score, Brier score, sensitivity, specificity, positive predictive value, negative predictive value, receiver operating characteristic curve, calibration curve, and decision curve analysis curve. For the selected optimal model, the decision-making process was interpreted using the SHapley Additive exPlanations (SHAP) method. The results show that the optimal RFE-XGBoost model achieved an accuracy of 0.83 (95% CI 0.76-0.90), an F1 score of 0.81 (95% CI 0.72-0.88), and an area under the receiver operating characteristic curve of 0.89 (95% CI 0.82-0.94). Furthermore, the model exhibited superior calibration and clinical utility. SHAP analysis revealed that increased perioperative transfusion quantity, higher tumor AJCC stage, elevated carcinoembryonic antigen level, elevated carbohydrate antigen 19-9 (CA19-9) level, advanced age, and elevated carbohydrate antigen 125 (CA125) level were correlated with increased individual mortality risk. The RFE-XGBoost model demonstrated excellent performance in predicting CRC patient prognosis, and the application of the SHAP method bolstered the model's credibility and utility.

Challenges and applications of artificial intelligence in infectious diseases and antimicrobial resistance

Artificial intelligence (AI) has transformed infectious disease control, enhancing rapid diagnosis and antibiotic discovery. While conventional tests delay diagnosis, AI-driven methods like machine learning and deep learning assist in pathogen detection, resistance prediction, and drug discovery. These tools improve antibiotic stewardship and identify effective compounds such as antimicrobial peptides and small molecules. This review explores AI applications in diagnostics, therapy, and drug discovery, emphasizing both strengths and areas needing improvement.

Benefits and harms associated with the use of AI-related algorithmic decision-making systems by healthcare professionals: a systematic review

Background

Despite notable advancements in artificial intelligence (AI) that enable complex systems to perform certain tasks more accurately than medical experts, the impact on patient-relevant outcomes remains uncertain. To address this gap, this systematic review assesses the benefits and harms associated with AI-related algorithmic decision-making (ADM) systems used by healthcare professionals, compared to standard care.

Methods

In accordance with the PRISMA guidelines, we included interventional and observational studies published as peer-reviewed full-text articles that met the following criteria: human patients; interventions involving algorithmic decision-making systems, developed with and/or utilizing machine learning (ML); and outcomes describing patient-relevant benefits and harms that directly affect health and quality of life, such as mortality and morbidity. Studies that did not undergo preregistration, lacked a standard-of-care control, or pertained to systems that assist in the execution of actions (e.g., in robotics) were excluded. We searched MEDLINE, EMBASE, IEEE Xplore, and Google Scholar for studies published in the past decade up to 31 March 2024. We assessed risk of bias using Cochrane's RoB 2 and ROBINS-I tools, and reporting transparency with CONSORT-AI and TRIPOD-AI. Two researchers independently managed the processes and resolved conflicts through discussion. This review has been registered with PROSPERO (CRD42023412156) and the study protocol has been published.

Findings

Out of 2,582 records identified after deduplication, 18 randomized controlled trials (RCTs) and one cohort study met the inclusion criteria, covering specialties such as psychiatry, oncology, and internal medicine. Collectively, the studies included a median of 243 patients (IQR 124-828), with a median of 50.5% female participants (range 12.5-79.0, IQR 43.6-53.6) across intervention and control groups. Four studies were classified as having low risk of bias, seven showed some concerns, and another seven were assessed as having high or serious risk of bias. Reporting transparency varied considerably: six studies showed high compliance, four moderate, and five low compliance with CONSORT-AI or TRIPOD-AI. Twelve studies (63%) reported patient-relevant benefits. Of those with low risk of bias, interventions reduced length of stay in hospital and intensive care unit (10.3 vs. 13.0 days, p = 0.042; 6.3 vs. 8.4 days, p = 0.030), in-hospital mortality (9.0% vs. 21.3%, p = 0.018), and depression symptoms in non-complex cases (45.1% vs. 52.3%, p = 0.03). However, harms were frequently underreported, with only eight studies (42%) documenting adverse events. No study reported an increase in adverse events as a result of the interventions.

Interpretation

The current evidence on AI-related ADM systems provides limited insights into patient-relevant outcomes. Our findings underscore the essential need for rigorous evaluations of clinical benefits, reinforced compliance with methodological standards, and balanced consideration of both benefits and harms to ensure meaningful integration into healthcare practice.

Funding

This study did not receive any funding.

Explainable Machine Learning Models for Brain Diseases: Insights from a Systematic Review

In recent years, Artificial Intelligence (AI) methods, specifically Machine Learning (ML) models, have been providing outstanding results in different areas of knowledge, with the health area being one of its most impactful fields of application. However, to be applied reliably, these models must provide users with clear, simple, and transparent explanations about the medical decision-making process. This systematic review aims to investigate the use and application of explainability in ML models used in brain disease studies. A systematic search was conducted in three major bibliographic databases, Web of Science, Scopus, and PubMed, from January 2014 to December 2023. A total of 133 relevant studies were identified and analyzed out of a total of 682 found in the initial search, in which the explainability of ML models in the medical context was studied, identifying 11 ML models and 12 explainability techniques applied in the study of 20 brain diseases.

Using real-world electronic health record data to predict the development of 12 cancer-related symptoms in the context of multimorbidity

Objective

This study uses electronic health record (EHR) data to predict 12 common cancer symptoms, assessing the efficacy of machine learning (ML) models in identifying symptom influencers.

Materials and Methods

We analyzed EHR data of 8156 adults diagnosed with cancer who underwent cancer treatment from 2017 to 2020. Structured and unstructured EHR data were sourced from the Enterprise Data Warehouse for Research at the University of Iowa Hospital and Clinics. Several predictive models, including logistic regression, random forest (RF), and XGBoost, were employed to forecast symptom development. The performances of the models were evaluated by F1-score and area under the curve (AUC) on the testing set. The SHapley Additive exPlanations framework was used to interpret these models and identify the predictive risk factors associated with fatigue as an exemplar.

Results

The RF model exhibited superior performance with a macro average AUC of 0.755 and an F1-score of 0.729 in predicting a range of cancer-related symptoms. For instance, the RF model achieved an AUC of 0.954 and an F1-score of 0.914 for pain prediction. Key predictive factors identified included clinical history, cancer characteristics, treatment modalities, and patient demographics depending on the symptom. For example, the odds ratio (OR) for fatigue was significantly influenced by allergy (OR = 2.3, 95% CI: 1.8-2.9) and colitis (OR = 1.9, 95% CI: 1.5-2.4).

Discussion

Our research emphasizes the critical integration of multimorbidity and patient characteristics in modeling cancer symptoms, revealing the considerable influence of chronic conditions beyond cancer itself.

Conclusion

We highlight the potential of ML for predicting cancer symptoms, suggesting a pathway for integrating such models into clinical systems to enhance personalized care and symptom management.

Large language models in plant biology

Large language models (LLMs), such as ChatGPT, have taken the world by storm. However, LLMs are not limited to human language and can be used to analyze sequential data, such as DNA, protein, and gene expression. The resulting foundation models can be repurposed to identify the complex patterns within the data, resulting in powerful, multipurpose prediction tools able to predict the state of cellular systems. This review outlines the different types of LLMs and showcases their recent uses in biology. Since LLMs have not yet been embraced by the plant community, we also cover how these models can be deployed for the plant kingdom.

Obtaining the Most Accurate, Explainable Model for Predicting Chronic Obstructive Pulmonary Disease: Triangulation of Multiple Linear Regression and Machine Learning Methods

BACKGROUND

Lung disease is a severe problem in the United States. Despite the decreasing rates of cigarette smoking, chronic obstructive pulmonary disease (COPD) continues to be a health burden in the United States. In this paper, we focus on COPD in the United States from 2016 to 2019.

OBJECTIVE

We gathered a diverse set of non-personally identifiable information from public data sources to better understand and predict COPD rates at the core-based statistical area (CBSA) level in the United States. Our objective was to compare linear models with machine learning models to obtain the most accurate and interpretable model of COPD.

METHODS

We integrated non-personally identifiable information from multiple Centers for Disease Control and Prevention sources and used them to analyze COPD with different types of methods. We included cigarette smoking, a well-known contributing factor, and race/ethnicity because health disparities among different races and ethnicities in the United States are also well known. The models also included the air quality index, education, employment, and economic variables. We fitted models with both multiple linear regression and machine learning methods.

RESULTS

The most accurate multiple linear regression model has variance explained of 81.1%, mean absolute error of 0.591, and symmetric mean absolute percentage error of 9.666. The most accurate machine learning model has variance explained of 85.7%, mean absolute error of 0.456, and symmetric mean absolute percentage error of 6.956. Overall, cigarette smoking and household income are the strongest predictor variables. Moderately strong predictors include education level and unemployment level, as well as American Indian or Alaska Native, Black, and Hispanic population percentages, all measured at the CBSA level.

CONCLUSIONS

This research highlights the importance of using diverse data sources as well as multiple methods to understand and predict COPD. The most accurate model was a gradient boosted tree, which captured nonlinearities in a model whose accuracy is superior to the best multiple linear regression. Our interpretable models suggest ways that individual predictor variables can be used in tailored interventions aimed at decreasing COPD rates in specific demographic and ethnographic communities. Gaps in understanding the health impacts of poor air quality, particularly in relation to climate change, suggest a need for further research to design interventions and improve public health.

Evaluating Explanations From AI Algorithms for Clinical Decision-Making: A Social Science-Based Approach

Explainable Artificial Intelligence (XAI) techniques generate explanations for predictions from AI models. These explanations can be evaluated for (i) faithfulness to the prediction, i.e., its correctness about the reasons for prediction, and (ii) usefulness to the user. While there are metrics to evaluate faithfulness, to our knowledge, there are no automated metrics to evaluate the usefulness of explanations in the clinical context. Our objective is to develop a new metric to evaluate usefulness of AI explanations to clinicians. Usefulness evaluation needs to consider both (a) how humans generally process explanations and (b) clinicians' specific requirements from explanations presented by clinical decision support systems (CDSS). Our new scoring method can evaluate the usefulness of explanations generated by any XAI method that provides importance values for the input features of the prediction model. Our method draws on theories from social science to gauge usefulness, and uses literature-derived biomedical knowledge graphs to quantify support for the explanations from clinical literature. We evaluate our method in a case study on predicting onset of sepsis in intensive care units. Our analysis shows that the scores obtained using our method corroborate with independent evidence from clinical literature and have the required qualities expected from such a metric. Thus, our method can be used to evaluate and select useful explanations from a diverse set of XAI techniques in clinical contexts, making it a fundamental tool for future research in the design of AI-driven CDSS.

Reliable automatic sleep stage classification based on hybrid intelligence

Sleep staging is a vital aspect of sleep assessment, serving as a critical tool for evaluating the quality of sleep and identifying sleep disorders. Manual sleep staging is a laborious process, while automatic sleep staging is seldom utilized in clinical practice due to issues related to the inadequate accuracy and interpretability of classification results in automatic sleep staging models. In this work, a hybrid intelligent model is presented for automatic sleep staging, which integrates data intelligence and knowledge intelligence, to attain a balance between accuracy, interpretability, and generalizability in the sleep stage classification. Specifically, it is built on any combination of typical electroencephalography (EEG) and electrooculography (EOG) channels, including a temporal fully convolutional network based on the U-Net architecture and a multi-task feature mapping structure. The experimental results show that, compared to current interpretable automatic sleep staging models, our model achieves a Macro-F1 score of 0.804 on the ISRUC dataset and 0.780 on the Sleep-EDFx dataset. Moreover, we use knowledge intelligence to address issues of excessive jumps and unreasonable sleep stage transitions in the coarse sleep graphs obtained by the model. We also explore the different ways knowledge intelligence affects coarse sleep graphs by combining different sleep graph correction methods. Our research can offer convenient support for sleep physicians, indicating its significant potential in improving the efficiency of clinical sleep staging.

Inference of social cognition in schizophrenia patients with neurocognitive domains and neurocognitive tests using automated machine learning

AIM

It has been suggested that single neurocognitive domain or neurocognitive test can be used to determine the overall cognitive function in schizophrenia using machine learning algorithms. It is unknown whether social cognition in schizophrenia patients can be estimated with machine learning based on neurocognitive domains or neurocognitive tests.

METHODS

To predict social cognition in schizophrenia, we applied an automated machine learning (AutoML) framework resulting from the analysis of predictive factors such as six neurocognitive domain scores and nine neurocognitive test scores of 380 schizophrenia patients in the Taiwanese population. Four clinical parameters (i.e., age, gender, subgroup, and education) were also used as predictive factors. We utilized an AutoML framework called Tree-based Pipeline Optimization Tool (TPOT) to generate predictive pipelines automatically.

RESULTS

The analysis revealed that all neurocognitive domains and tests except the reasoning and problem solving domain/test showed significant associations with social cognition. In addition, a TPOT-generated pipeline can best predict social cognition in schizophrenia using seven predictive factors, including five neurocognitive domains (i.e., speed of processing, sustained attention, working memory, verbal learning and memory, and visual learning and memory) and two clinical parameters (i.e., age and gender). This predictive pipeline consists of machine learning algorithms such as function transformers, an approximate feature map, independent component analysis, and linear regression.

CONCLUSION

The study indicates that an AutoML framework such as TPOT may provide a promising way to produce truly effective machine learning pipelines for predicting social cognition in schizophrenia using neurocognitive domains and/or neurocognitive tests.

The shape of cancer relapse: Topological data analysis predicts recurrence in paediatric acute lymphoblastic leukaemia

Although children and adolescents with acute lymphoblastic leukaemia (ALL) have high survival rates, approximately 15-20% of patients relapse. Risk of relapse is routinely estimated at diagnosis by biological factors, including flow cytometry data. This high-dimensional data is typically manually assessed by projecting it onto a subset of biomarkers. Cell density and "empty spaces" in 2D projections of the data, i.e. regions devoid of cells, are then used for qualitative assessment. Here, we use topological data analysis (TDA), which quantifies shapes, including empty spaces, in data, to analyse pre-treatment ALL datasets with known patient outcomes. We combine these fully unsupervised analyses with Machine Learning (ML) to identify significant shape characteristics and demonstrate that they accurately predict risk of relapse, particularly for patients previously classified as 'low risk'. We independently confirm the predictive power of CD10, CD20, CD38, and CD45 as biomarkers for ALL diagnosis. Based on our analyses, we propose three increasingly detailed prognostic pipelines for analysing flow cytometry data from ALL patients depending on technical and technological availability: 1. Visual inspection of specific biological features in biparametric projections of the data; 2. Computation of quantitative topological descriptors of such projections; 3. A combined analysis, using TDA and ML, in the four-parameter space defined by CD10, CD20, CD38 and CD45. Our analyses readily extend to other haematological malignancies.

Quantification of spatial and phenotypic heterogeneity in an agent-based model of tumour-macrophage interactions

We introduce a new spatial statistic, the weighted pair correlation function (wPCF). The wPCF extends the existing pair correlation function (PCF) and cross-PCF to describe spatial relationships between points marked with combinations of discrete and continuous labels. We validate its use through application to a new agent-based model (ABM) which simulates interactions between macrophages and tumour cells. These interactions are influenced by the spatial positions of the cells and by macrophage phenotype, a continuous variable that ranges from anti-tumour to pro-tumour. By varying model parameters that regulate macrophage phenotype, we show that the ABM exhibits behaviours which resemble the 'three Es of cancer immunoediting': Equilibrium, Escape, and Elimination. We use the wPCF to analyse synthetic images generated by the ABM. We show that the wPCF generates a 'human readable' statistical summary of where macrophages with different phenotypes are located relative to both blood vessels and tumour cells. We also define a distinct 'PCF signature' that characterises each of the three Es of immunoediting, by combining wPCF measurements with the cross-PCF describing interactions between vessels and tumour cells. By applying dimension reduction techniques to this signature, we identify its key features and train a support vector machine classifier to distinguish between simulation outputs based on their PCF signature. This proof-of-concept study shows how multiple spatial statistics can be combined to analyse the complex spatial features that the ABM generates, and to partition them into interpretable groups. The intricate spatial features produced by the ABM are similar to those generated by state-of-the-art multiplex imaging techniques which distinguish the spatial distribution and intensity of multiple biomarkers in biological tissue regions. Applying methods such as the wPCF to multiplex imaging data would exploit the continuous variation in biomarker intensities and generate more detailed characterisation of the spatial and phenotypic heterogeneity in tissue samples.

On the importance of interpretable machine learning predictions to inform clinical decision making in oncology

Machine learning-based tools are capable of guiding individualized clinical management and decision-making by providing predictions of a patient's future health state. Through their ability to model complex nonlinear relationships, ML algorithms can often outperform traditional statistical prediction approaches, but the use of nonlinear functions can mean that ML techniques may also be less interpretable than traditional statistical methodologies. While there are benefits of intrinsic interpretability, many model-agnostic approaches now exist and can provide insight into the way in which ML systems make decisions. In this paper, we describe how different algorithms can be interpreted and introduce some techniques for interpreting complex nonlinear algorithms.

A novel method to derive personalized minimum viable recommendations for type 2 diabetes prevention based on counterfactual explanations

Despite the growing availability of artificial intelligence models for predicting type 2 diabetes, there is still a lack of personalized approaches to quantify minimum viable changes in biomarkers that may help reduce the individual risk of developing disease. The aim of this article is to develop a new method, based on counterfactual explanations, to generate personalized recommendations to reduce the one-year risk of type 2 diabetes. Ten routinely collected biomarkers extracted from Electronic Medical Records of 2791 patients at low risk and 2791 patients at high risk of type 2 diabetes were analyzed. Two regions characterizing the two classes of patients were estimated using a Support Vector Data Description classifier. Counterfactual explanations (i.e., minimal changes in input features able to change the risk class) were generated for patients at high risk and evaluated using performance metrics (availability, validity, actionability, similarity, and discriminative power) and a qualitative survey administered to seven expert clinicians. Results showed that, on average, the requested minimum viable changes implied a significant reduction of fasting blood sugar, systolic blood pressure, and triglycerides and a significant increase of high-density lipoprotein in patients at risk of diabetes. A significant reduction in body mass index was also recommended in most of the patients at risk, except in females without hypertension. In general, greater changes were recommended in hypertensive patients compared to non-hypertensive ones. The experts were overall satisfied with the proposed approach although in some cases the proposed recommendations were deemed insufficient to reduce the risk in a clinically meaningful way. Future research will focus on a larger set of biomarkers and different comorbidities, also incorporating clinical guidelines whenever possible. Development of additional mathematical and clinical validation approaches will also be of paramount importance.