Empirical assessment of bias in machine learning diagnostic test accuracy studies

Screening for obstructive sleep apnea in patients with cancer - a machine learning approach

Background

Obstructive sleep apnea (OSA) is a highly prevalent sleep disorder associated with daytime sleepiness, fatigue, and increased all-cause mortality risk in patients with cancer. Existing screening tools for OSA do not account for the interaction of cancer-related features that may increase OSA risk.

Study Design and Methods

This is a retrospective study of patients with cancer at a single tertiary cancer institution who underwent a home sleep apnea test (HSAT) to evaluate for OSA. Unsupervised machine learning (ML) was used to reduce the dimensions and extract significant features associated with OSA. ML classifiers were applied to principal components and model hyperparameters were optimized using k-fold cross-validation. Training models for OSA were subsequently tested and compared with the STOP-Bang questionnaire on a prospective unseen test set of patients who underwent an HSAT.

Results

From a training dataset of 249 patients, kernel principal component analysis (PCA) extracted eight components through dimension reduction to explain the maximum variance with OSA at 98%. Predictors of OSA were smoking, asthma, chronic kidney disease, STOP-Bang score, race, diabetes, radiation to head/neck/thorax (RT-HNT), type of cancer, and cancer metastases. Of the ML models, PCA + RF had the highest sensitivity (96.8%), specificity (92.3%), negative predictive value (92%), F1 score (0.93), and ROC-AUC score (0.88). The PCA + RF screening algorithm also performed better than the STOP-Bang questionnaire alone when tested on a prospective unseen test set.

Conclusions

The PCA + RF ML model had the highest accuracy in screening for OSA in patients with cancer. History of RT-HNT, cancer metastases, and type of cancer were identified as cancer-related risk factors for OSA.

Extracting cancer concepts from clinical notes using natural language processing: a systematic review

BACKGROUND

Extracting information from free texts using natural language processing (NLP) can save time and reduce the hassle of manually extracting large quantities of data from incredibly complex clinical notes of cancer patients. This study aimed to systematically review studies that used NLP methods to identify cancer concepts from clinical notes automatically.

METHODS

PubMed, Scopus, Web of Science, and Embase were searched for English language papers using a combination of the terms concerning "Cancer", "NLP", "Coding", and "Registries" until June 29, 2021. Two reviewers independently assessed the eligibility of papers for inclusion in the review.

RESULTS

Most of the software programs used for concept extraction reported were developed by the researchers (n = 7). Rule-based algorithms were the most frequently used algorithms for developing these programs. In most articles, the criteria of accuracy (n = 14) and sensitivity (n = 12) were used to evaluate the algorithms. In addition, Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT) and Unified Medical Language System (UMLS) were the most commonly used terminologies to identify concepts. Most studies focused on breast cancer (n = 4, 19%) and lung cancer (n = 4, 19%).

CONCLUSION

The use of NLP for extracting the concepts and symptoms of cancer has increased in recent years. The rule-based algorithms are well-liked algorithms by developers. Due to these algorithms' high accuracy and sensitivity in identifying and extracting cancer concepts, we suggested that future studies use these algorithms to extract the concepts of other diseases as well.

Clinical Informatics Approaches to Understand and Address Cancer Disparities

OBJECTIVES

Disparities in cancer incidence and outcomes across race, ethnicity, gender, socioeconomic status, and geography are well-documented, but their etiologies are often poorly understood and multifactorial. Clinical informatics can provide tools to better understand and address these disparities by enabling high-throughput analysis of multiple types of data. Here, we review recent efforts in clinical informatics to study and measure disparities in cancer.

METHODS

We carried out a narrative review of clinical informatics studies related to cancer disparities and bias published from 2018-2021, with a focus on domains such as real-world data (RWD) analysis, natural language processing (NLP), radiomics, genomics, proteomics, metabolomics, and metagenomics.

RESULTS

Clinical informatics studies that investigated cancer disparities across race, ethnicity, gender, and age were identified. Most cancer disparities work within clinical informatics used RWD analysis, NLP, radiomics, and genomics. Emerging applications of clinical informatics to understand cancer disparities, including proteomics, metabolomics, and metagenomics, were less well represented in the literature but are promising future research avenues. Algorithmic bias was identified as an important consideration when developing and implementing cancer clinical informatics techniques, and efforts to address this bias were reviewed.

CONCLUSIONS

In recent years, clinical informatics has been used to probe a range of data sources to understand cancer disparities across different populations. As informatics tools become integrated into clinical decision-making, attention will need to be paid to ensure that algorithmic bias does not amplify existing disparities. In our increasingly interconnected medical systems, clinical informatics is poised to untap the full potential of multi-platform health data to address cancer disparities.

Trial-level characteristics associate with treatment effect estimates: a systematic review of meta-epidemiological studies

BACKGROUND

To summarize the up-to-date empirical evidence on trial-level characteristics of randomized controlled trials associated with treatment effect estimates.

METHODS

A systematic review searched three databases up to August 2020. Meta-epidemiological (ME) studies of randomized controlled trials on intervention effect were eligible. We assessed the methodological quality of ME studies using a self-developed criterion. Associations between treatment effect estimates and trial-level characteristics were presented using forest plots.

RESULTS

Eighty ME studies were included, with 25/80 (31%) being published after 2015. Less than one-third ME studies critically appraised the included studies (26/80, 33%), published a protocol (23/80, 29%), and provided a list of excluded studies with justifications (12/80, 15%). Trials with high or unclear (versus low) risk of bias on sequence generation (3/14 for binary outcome and 1/6 for continuous outcome), allocation concealment (11/18 and 1/6), double blinding (5/15 and 2/4) and smaller sample size (4/5 and 2/2) significantly associated with larger treatment effect estimates. Associations between high or unclear risk of bias on allocation concealment (5/6 for binary outcome and 1/3 for continuous outcome), double blinding (4/5 and 1/3) and larger treatment effect estimates were more frequently observed for subjective outcomes. The associations between treatment effect estimates and non-blinding of outcome assessors were removed in trials using multiple observers to reach consensus for both binary and continuous outcomes. Some trial characteristics in the Cochrane risk-of-bias (RoB2) tool have not been covered by the included ME studies, including using validated method for outcome measures and selection of the reported results from multiple outcome measures or multiple analysis based on results (e.g., significance of the results).

CONCLUSIONS

Consistently significant associations between larger treatment effect estimates and high or unclear risk of bias on sequence generation, allocation concealment, double blinding and smaller sample size were found. The impact of allocation concealment and double blinding were more consistent for subjective outcomes. The methodological and reporting quality of included ME studies were dissatisfactory. Future ME studies should follow the corresponding reporting guideline. Specific guidelines for conducting and critically appraising ME studies are needed.

Machine Learning Decomposition of the Anatomy of Neuropsychological Deficit in Alzheimer’s Disease and Mild Cognitive Impairment

Objective

Alzheimer’s Disease (AD) is a progressive condition characterized by cognitive decline. AD is often preceded by mild cognitive impairment (MCI), though the diagnosis of both conditions remains a challenge. Early diagnosis of AD, and prediction of MCI progression require data-driven approaches to improve patient selection for treatment. We used a machine learning tool to predict performance in neuropsychological tests in AD and MCI based on functional connectivity using a whole-brain connectome, in an attempt to identify network substrates of cognitive deficits in AD.

Methods

Neuropsychological tests, baseline anatomical T1 magnetic resonance imaging (MRI), resting-state functional MRI, and diffusion weighted imaging scans were obtained from 149 MCI, and 85 AD patients; and 140 cognitively unimpaired geriatric participants. A novel machine learning tool, Hollow Tree Super (HoTS) was utilized to extract feature importance from each machine learning model to identify brain regions that were associated with deficit and absence of deficit for 11 neuropsychological tests.

Results

11 models attained an area under the receiver operating curve (AUC-ROC) greater than 0.65, while five models had an AUC-ROC ≥ 0.7. 20 parcels of the Human Connectome Project Multimodal Parcelation Atlas matched to poor performance in at least two neuropsychological tests, while 14 parcels were associated with good performance in at least two tests. At a network level, most parcels predictive of both presence and absence of deficit were affiliated with the Central Executive Network, Default Mode Network, and the Sensorimotor Networks. Segregating predictors by the cognitive domain associated with each test revealed areas of coherent overlap between cognitive domains, with the parcels providing possible markers to screen for cognitive impairment.

Conclusion

Approaches such as ours which incorporate whole-brain functional connectivity and harness feature importance in machine learning models may aid in identifying diagnostic and therapeutic targets in AD.

Clinician Preimplementation Perspectives of a Decision-Support Tool for the Prediction of Cardiac Arrhythmia Based on Machine Learning: Near-Live Feasibility and Qualitative Study

BACKGROUND

Artificial intelligence (AI), such as machine learning (ML), shows great promise for improving clinical decision-making in cardiac diseases by outperforming statistical-based models. However, few AI-based tools have been implemented in cardiology clinics because of the sociotechnical challenges during transitioning from algorithm development to real-world implementation.

OBJECTIVE

This study explored how an ML-based tool for predicting ventricular tachycardia and ventricular fibrillation (VT/VF) could support clinical decision-making in the remote monitoring of patients with an implantable cardioverter defibrillator (ICD).

METHODS

Seven experienced electrophysiologists participated in a near-live feasibility and qualitative study, which included walkthroughs of 5 blinded retrospective patient cases, use of the prediction tool, and questionnaires and interview questions. All sessions were video recorded, and sessions evaluating the prediction tool were transcribed verbatim. Data were analyzed through an inductive qualitative approach based on grounded theory.

RESULTS

The prediction tool was found to have potential for supporting decision-making in ICD remote monitoring by providing reassurance, increasing confidence, acting as a second opinion, reducing information search time, and enabling delegation of decisions to nurses and technicians. However, the prediction tool did not lead to changes in clinical action and was found less useful in cases where the quality of data was poor or when VT/VF predictions were found to be irrelevant for evaluating the patient.

CONCLUSIONS

When transitioning from AI development to testing its feasibility for clinical implementation, we need to consider the following: expectations must be aligned with the intended use of AI; trust in the prediction tool is likely to emerge from real-world use; and AI accuracy is relational and dependent on available information and local workflows. Addressing the sociotechnical gap between the development and implementation of clinical decision-support tools based on ML in cardiac care is essential for succeeding with adoption. It is suggested to include clinical end-users, clinical contexts, and workflows throughout the overall iterative approach to design, development, and implementation.

Artificial intelligence applications in different imaging modalities for corneal topography

Interpretation of topographical maps used to detect corneal ectasias requires a high level of expertise. Several artificial intelligence (AI) technologies have attempted to interpret topographic maps. The purpose of this study is to provide a review of AI algorithms in corneal topography from the perspectives of an eye care professional, a biomedical engineer, and a data scientist. A systematic literature review using Web of Science, Pubmed, and Google Scholar was performed from 2010 to 2020 on themes regarding imaging modalities, their parameters, purpose, and conclusions and their samples and performance related to AI in corneal topography. We provide a comprehensive summary of advances in corneal imaging and its applications in AI. Combined metrics from the Dual Scheimpflug and Placido device could be a good starting point to try AI models in corneal imaging systems. The range of area under the receiving operating curve for AI in keratoconus detection and classification was from 0.87 to 1, sensitivity was from 0.89 to 1, and specificity was from 0.82 to 1. A combination of different types of AI applications to corneal ectasia diagnosis is recommended.

Detection and Evaluation of Machine Learning Bias

Machine learning models are built using training data, which is collected from human experience and is prone to bias. Humans demonstrate a cognitive bias in their thinking and behavior, which is ultimately reflected in the collected data. From Amazon’s hiring system, which was built using ten years of human hiring experience, to a judicial system that was trained using human judging practices, these systems all include some element of bias. The best machine learning models are said to mimic humans’ cognitive ability, and thus such models are also inclined towards bias. However, detecting and evaluating bias is a very important step for better explainable models. In this work, we aim to explain bias in learning models in relation to humans’ cognitive bias and propose a wrapper technique to detect and evaluate bias in machine learning models using an openly accessible dataset from UCI Machine Learning Repository. In the deployed dataset, the potentially biased attributes (PBAs) are gender and race. This study introduces the concept of alternation functions to swap the values of PBAs, and evaluates the impact on prediction using KL divergence. Results demonstrate females and Asians to be associated with low wages, placing some open research questions for the research community to ponder over.

Enhancing trust in AI through industry self-governance

Artificial intelligence (AI) is critical to harnessing value from exponentially growing health and healthcare data. Expectations are high for AI solutions to effectively address current health challenges. However, there have been prior periods of enthusiasm for AI followed by periods of disillusionment, reduced investments, and progress, known as "AI Winters." We are now at risk of another AI Winter in health/healthcare due to increasing publicity of AI solutions that are not representing touted breakthroughs, and thereby decreasing trust of users in AI. In this article, we first highlight recently published literature on AI risks and mitigation strategies that would be relevant for groups considering designing, implementing, and promoting self-governance. We then describe a process for how a diverse group of stakeholders could develop and define standards for promoting trust, as well as AI risk-mitigating practices through greater industry self-governance. We also describe how adherence to such standards could be verified, specifically through certification/accreditation. Self-governance could be encouraged by governments to complement existing regulatory schema or legislative efforts to mitigate AI risks. Greater adoption of industry self-governance could fill a critical gap to construct a more comprehensive approach to the governance of AI solutions than US legislation/regulations currently encompass. In this more comprehensive approach, AI developers, AI users, and government/legislators all have critical roles to play to advance practices that maintain trust in AI and prevent another AI Winter.

Predicting breast cancer 5-year survival using machine learning: A systematic review

BACKGROUND

Accurately predicting the survival rate of breast cancer patients is a major issue for cancer researchers. Machine learning (ML) has attracted much attention with the hope that it could provide accurate results, but its modeling methods and prediction performance remain controversial. The aim of this systematic review is to identify and critically appraise current studies regarding the application of ML in predicting the 5-year survival rate of breast cancer.

METHODS

In accordance with the PRISMA guidelines, two researchers independently searched the PubMed (including MEDLINE), Embase, and Web of Science Core databases from inception to November 30, 2020. The search terms included breast neoplasms, survival, machine learning, and specific algorithm names. The included studies related to the use of ML to build a breast cancer survival prediction model and model performance that can be measured with the value of said verification results. The excluded studies in which the modeling process were not explained clearly and had incomplete information. The extracted information included literature information, database information, data preparation and modeling process information, model construction and performance evaluation information, and candidate predictor information.

RESULTS

Thirty-one studies that met the inclusion criteria were included, most of which were published after 2013. The most frequently used ML methods were decision trees (19 studies, 61.3%), artificial neural networks (18 studies, 58.1%), support vector machines (16 studies, 51.6%), and ensemble learning (10 studies, 32.3%). The median sample size was 37256 (range 200 to 659820) patients, and the median predictor was 16 (range 3 to 625). The accuracy of 29 studies ranged from 0.510 to 0.971. The sensitivity of 25 studies ranged from 0.037 to 1. The specificity of 24 studies ranged from 0.008 to 0.993. The AUC of 20 studies ranged from 0.500 to 0.972. The precision of 6 studies ranged from 0.549 to 1. All of the models were internally validated, and only one was externally validated.

CONCLUSIONS

Overall, compared with traditional statistical methods, the performance of ML models does not necessarily show any improvement, and this area of research still faces limitations related to a lack of data preprocessing steps, the excessive differences of sample feature selection, and issues related to validation. Further optimization of the performance of the proposed model is also needed in the future, which requires more standardization and subsequent validation.

Machine Learning Models for Predicting Neonatal Mortality: A Systematic Review

INTRODUCTION

Approximately 7,000 newborns die every day, accounting for almost half of child deaths under 5 years of age. Deciphering which neonates are at increased risk for mortality can have an important global impact. As such, integrating high computational technology (e.g., artificial intelligence [AI]) may help identify the early and potentially modifiable predictors of neonatal mortality. Therefore, the objective of this study was to collate, critically appraise, and analyze neonatal prediction studies that included AI.

METHODS

A literature search was performed in PubMed, Cochrane, OVID, and Google Scholar. We included studies that used AI (e.g., machine learning (ML) and deep learning) to formulate prediction models for neonatal death. We excluded small studies (n < 500 individuals) and studies using only antenatal factors to predict mortality. Two independent investigators screened all articles for inclusion. The data collection consisted of study design, number of models, features used per model, feature importance, internal and/or external validation, and calibration analysis. Our primary outcome was the average area under the receiving characteristic curve (AUC) or sensitivity and specificity for all models included in each study.

RESULTS

Of 434 articles, 11 studies were included. The total number of participants was 1.26 M with gestational ages ranging from 22 weeks to term. Number of features ranged from 3 to 66 with timing of prediction as early as 5 min of life to a maximum of 7 days of age. The average number of models per study was 4, with neural network, random forest, and logistic regression comprising the most used models (58.3%). Five studies (45.5%) reported calibration plots and 2 (18.2%) conducted external validation. Eight studies reported results by AUC and 5 studies reported the sensitivity and specificity. The AUC varied from 58.3% to 97.0%. The mean sensitivities ranged from 63% to 80% and specificities from 78% to 99%. The best overall model was linear discriminant analysis, but it also had a high number of features (n = 17).

DISCUSSION/CONCLUSION

ML models can accurately predict death in neonates. This analysis demonstrates the most commonly used predictors and metrics for AI prediction models for neonatal mortality. Future studies should focus on external validation, calibration, as well as deployment of applications that can be readily accessible to health-care providers.

Extracellular MicroRNAs as Intercellular Mediators and Noninvasive Biomarkers of Cancer

Simple Summary

There are an extensive number of publications regarding the role of endogenous miRNAs as regulators of gene expression in cancer. However, extracellular miRNAs have emerged as a novel mechanism of cell-to-cell communication in normal conditions and disease and have drawn a large amount of interest as regulators of gene expression and as potential non-invasive biomarkers in cancer. Despite this high interest and the abundance of research on the biology and role of extracellular miRNAs in cancer, they are not yet completely understood. The aim of this review is to highlight the relevant biological characteristics of extracellular miRNAs that enable them to function as intercellular mediators of gene expression regulation and provide the recently published evidence of the specific role of extracellular miRNAs in tumor development and progression.

Abstract

MicroRNAs (miRNAs) are released by different types of cells through highly regulated mechanisms under normal and pathological conditions. These extracellular miRNAs can be delivered into recipient cells for functional purposes, acting as cell-to-cell signaling mediators. It has been discovered that cancer cells release miRNAs into their surroundings, targeting normal cells or other cancer cells, presumably to promote tumor development and progression. These extracellular miRNAs are associated with oncogenic mechanisms and, because they can be quantified in blood and other bodily fluids, may be suitable noninvasive biomarkers for cancer detection. This review summarizes recent evidence of the role of extracellular miRNAs as intercellular mediators, with an emphasis on their role in the mechanisms of tumor development and progression and their potential value as biomarkers in solid tumors. It also highlights the biological characteristics of extracellular miRNAs that enable them to function as regulators of gene expression, such as biogenesis, gene silencing mechanisms, subcellular compartmentalization, and the functions and mechanisms of release.