Bias in artificial intelligence algorithms and recommendations for mitigation

Variation in monitoring: Glucose measurement in the ICU as a case study to preempt spurious correlations

OBJECTIVE

Health inequities can be influenced by demographic factors such as race and ethnicity, proficiency in English, and biological sex. Disparities may manifest as differential likelihood of testing which correlates directly with the likelihood of an intervention to address an abnormal finding. Our retrospective observational study evaluated the presence of variation in glucose measurements in the Intensive Care Unit (ICU).

METHODS

Using the MIMIC-IV database (2008-2019), a single-center, academic referral hospital in Boston (USA), we identified adult patients meeting sepsis-3 criteria. Exclusion criteria were diabetic ketoacidosis, ICU length of stay under 1 day, and unknown race or ethnicity. We performed a logistic regression analysis to assess differential likelihoods of glucose measurements on day 1. A negative binomial regression was fitted to assess the frequency of subsequent glucose readings. Analyses were adjusted for relevant clinical confounders, and performed across three disparity proxy axes: race and ethnicity, sex, and English proficiency.

RESULTS

We studied 24,927 patients, of which 19.5% represented racial and ethnic minority groups, 42.4% were female, and 9.8% had limited English proficiency. No significant differences were found for glucose measurement on day 1 in the ICU. This pattern was consistent irrespective of the axis of analysis, i.e. race and ethnicity, sex, or English proficiency. Conversely, subsequent measurement frequency revealed potential disparities. Specifically, males (incidence rate ratio (IRR) 1.06, 95% confidence interval (CI) 1.01 - 1.21), patients who identify themselves as Hispanic (IRR 1.11, 95% CI 1.01 - 1.21), or Black (IRR 1.06, 95% CI 1.01 - 1.12), and patients being English proficient (IRR 1.08, 95% CI 1.01 - 1.15) had higher chances of subsequent glucose readings.

CONCLUSION

We found disparities in ICU glucose measurements among patients with sepsis, albeit the magnitude was small. Variation in disease monitoring is a source of data bias that may lead to spurious correlations when modeling health data.

AI and Ethics: A Systematic Review of the Ethical Considerations of Large Language Model Use in Surgery Research

INTRODUCTION

As large language models receive greater attention in medical research, the investigation of ethical considerations is warranted. This review aims to explore surgery literature to identify ethical concerns surrounding these artificial intelligence models and evaluate how autonomy, beneficence, nonmaleficence, and justice are represented within these ethical discussions to provide insights in order to guide further research and practice.

METHODS

A systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Five electronic databases were searched in October 2023. Eligible studies included surgery-related articles that focused on large language models and contained adequate ethical discussion. Study details, including specialty and ethical concerns, were collected.

RESULTS

The literature search yielded 1179 articles, with 53 meeting the inclusion criteria. Plastic surgery, orthopedic surgery, and neurosurgery were the most represented surgical specialties. Autonomy was the most explicitly cited ethical principle. The most frequently discussed ethical concern was accuracy (n = 45, 84.9%), followed by bias, patient confidentiality, and responsibility.

CONCLUSION

The ethical implications of using large language models in surgery are complex and evolving. The integration of these models into surgery necessitates continuous ethical discourse to ensure responsible and ethical use, balancing technological advancement with human dignity and safety.

Artificial intelligence in diagnosing upper limb musculoskeletal disorders: a systematic review and meta-analysis of diagnostic tests

Purpose

The integration of artificial intelligence (AI) in radiology has revolutionized diagnostics, optimizing precision and decision-making. Specifically in musculoskeletal imaging, AI tools can improve accuracy for upper extremity pathologies. This study aimed to assess the diagnostic performance of AI models in detecting musculoskeletal pathologies of the upper extremity using different imaging modalities.

Methods

A meta-analysis was conducted, involving searches on MEDLINE/PubMed, SCOPUS, Cochrane Library, Lilacs, and SciELO. The quality of the studies was assessed using the QUADAS-2 tool. Diagnostic accuracy measures including sensitivity, specificity, diagnostic odds ratio (DOR), positive and negative likelihood ratios (PLR, NLR), area under the curve (AUC), and summary receiver operating characteristic were pooled using a random-effects model. Heterogeneity and subgroup analyses were also included. All statistical analyses and plots were performed using the R software package.

Results

Thirteen models from ten articles were analyzed. The sensitivity and specificity of the AI models to detect musculoskeletal conditions in the upper extremity were 0.926 (95% CI: 0.900; 0.945) and 0.908 (95% CI: 0.810; 0.958). The PLR, NLR, lnDOR, and the AUC estimates were found to be 19.18 (95% CI: 8.90; 29.34), 0.11 (95% CI: 0.18; 0.46), 4.62 (95% CI: 4.02; 5.22) with a (P < 0.001), and 95%, respectively.

Conclusion

The AI models exhibited strong univariate and bivariate performance in detecting both positive and negative cases within the analyzed dataset of musculoskeletal pathologies in the upper extremity.

Algor-ethics: charting the ethical path for AI in critical care

The integration of Clinical Decision Support Systems (CDSS) based on artificial intelligence (AI) in healthcare is groundbreaking evolution with enormous potential, but its development and ethical implementation, presents unique challenges, particularly in critical care, where physicians often deal with life-threating conditions requiring rapid actions and patients unable to participate in the decisional process. Moreover, development of AI-based CDSS is complex and should address different sources of bias, including data acquisition, health disparities, domain shifts during clinical use, and cognitive biases in decision-making. In this scenario algor-ethics is mandatory and emphasizes the integration of 'Human-in-the-Loop' and 'Algorithmic Stewardship' principles, and the benefits of advanced data engineering. The establishment of Clinical AI Departments (CAID) is necessary to lead AI innovation in healthcare, ensuring ethical integrity and human-centered development in this rapidly evolving field.

Critical review of self-diagnosis of mental health conditions using artificial intelligence

The advent of artificial intelligence (AI) has revolutionised various aspects of our lives, including mental health nursing. AI-driven tools and applications have provided a convenient and accessible means for individuals to assess their mental well-being within the confines of their homes. Nonetheless, the widespread trend of self-diagnosing mental health conditions through AI poses considerable risks. This review article examines the perils associated with relying on AI for self-diagnosis in mental health, highlighting the constraints and possible adverse outcomes that can arise from such practices. It delves into the ethical, psychological, and social implications, underscoring the vital role of mental health professionals, including psychologists, psychiatrists, and nursing specialists, in providing professional assistance and guidance. This article aims to highlight the importance of seeking professional assistance and guidance in addressing mental health concerns, especially in the era of AI-driven self-diagnosis.

Traditional, complementary, and integrative medicine and artificial intelligence: Novel opportunities in healthcare

The convergence of traditional, complementary, and integrative medicine (TCIM) with artificial intelligence (AI) is a promising frontier in healthcare. TCIM is a patient-centric approach that combines conventional medicine with complementary therapies, emphasizing holistic well-being. AI can revolutionize healthcare through data-driven decision-making and personalized treatment plans. This article explores how AI technologies can complement and enhance TCIM, aligning with the shared objectives of researchers from both fields in improving patient outcomes, enhancing care quality, and promoting holistic wellness. This integration of TCIM and AI introduces exciting opportunities but also noteworthy challenges. AI may augment TCIM by assisting in early disease detection, providing personalized treatment plans, predicting health trends, and enhancing patient engagement. Challenges at the intersection of AI and TCIM include data privacy and security, regulatory complexities, maintaining the human touch in patient-provider relationships, and mitigating bias in AI algorithms. Patients' trust, informed consent, and legal accountability are all essential considerations. Future directions in AI-enhanced TCIM include advanced personalized medicine, understanding the efficacy of herbal remedies, and studying patient-provider interactions. Research on bias mitigation, patient acceptance, and trust in AI-driven TCIM healthcare is crucial. In this article, we outlined that the merging of TCIM and AI holds great promise in enhancing healthcare delivery, personalizing treatment plans, preventive care, and patient engagement. Addressing challenges and fostering collaboration between AI experts, TCIM practitioners, and policymakers, however, is vital to harnessing the full potential of this integration.

The Role of Artificial Intelligence in Improving Patient Outcomes and Future of Healthcare Delivery in Cardiology: A Narrative Review of the Literature

Cardiovascular diseases exert a significant burden on the healthcare system worldwide. This narrative literature review discusses the role of artificial intelligence (AI) in the field of cardiology. AI has the potential to assist healthcare professionals in several ways, such as diagnosing pathologies, guiding treatments, and monitoring patients, which can lead to improved patient outcomes and a more efficient healthcare system. Moreover, clinical decision support systems in cardiology have improved significantly over the past decade. The addition of AI to these clinical decision support systems can improve patient outcomes by processing large amounts of data, identifying subtle associations, and providing a timely, evidence-based recommendation to healthcare professionals. Lastly, the application of AI allows for personalized care by utilizing predictive models and generating patient-specific treatment plans. However, there are several challenges associated with the use of AI in healthcare. The application of AI in healthcare comes with significant cost and ethical considerations. Despite these challenges, AI will be an integral part of healthcare delivery in the near future, leading to personalized patient care, improved physician efficiency, and anticipated better outcomes.

Artificial Intelligence in Kidney Disease: A Comprehensive Study and Directions for Future Research

Artificial intelligence (AI) has emerged as a promising tool in the field of healthcare, with an increasing number of research articles evaluating its applications in the domain of kidney disease. To comprehend the evolving landscape of AI research in kidney disease, a bibliometric analysis is essential. The purposes of this study are to systematically analyze and quantify the scientific output, research trends, and collaborative networks in the application of AI to kidney disease. This study collected AI-related articles published between 2012 and 20 November 2023 from the Web of Science. Descriptive analyses of research trends in the application of AI in kidney disease were used to determine the growth rate of publications by authors, journals, institutions, and countries. Visualization network maps of country collaborations and author-provided keyword co-occurrences were generated to show the hotspots and research trends in AI research on kidney disease. The initial search yielded 673 articles, of which 631 were included in the analyses. Our findings reveal a noteworthy exponential growth trend in the annual publications of AI applications in kidney disease. Nephrology Dialysis Transplantation emerged as the leading publisher, accounting for 4.12% (26 out of 631 papers), followed by the American Journal of Transplantation at 3.01% (19/631) and Scientific Reports at 2.69% (17/631). The primary contributors were predominantly from the United States (n = 164, 25.99%), followed by China (n = 156, 24.72%) and India (n = 62, 9.83%). In terms of institutions, Mayo Clinic led with 27 contributions (4.27%), while Harvard University (n = 19, 3.01%) and Sun Yat-Sen University (n = 16, 2.53%) secured the second and third positions, respectively. This study summarized AI research trends in the field of kidney disease through statistical analysis and network visualization. The findings show that the field of AI in kidney disease is dynamic and rapidly progressing and provides valuable information for recognizing emerging patterns, technological shifts, and interdisciplinary collaborations that contribute to the advancement of knowledge in this critical domain.

Machine Learning and Bias in Medical Imaging: Opportunities and Challenges

Bias in health care has been well documented and results in disparate and worsened outcomes for at-risk groups. Medical imaging plays a critical role in facilitating patient diagnoses but involves multiple sources of bias including factors related to access to imaging modalities, acquisition of images, and assessment (ie, interpretation) of imaging data. Machine learning (ML) applied to diagnostic imaging has demonstrated the potential to improve the quality of imaging-based diagnosis and the precision of measuring imaging-based traits. Algorithms can leverage subtle information not visible to the human eye to detect underdiagnosed conditions or derive new disease phenotypes by linking imaging features with clinical outcomes, all while mitigating cognitive bias in interpretation. Importantly, however, the application of ML to diagnostic imaging has the potential to either reduce or propagate bias. Understanding the potential gain as well as the potential risks requires an understanding of how and what ML models learn. Common risks of propagating bias can arise from unbalanced training, suboptimal architecture design or selection, and uneven application of models. Notwithstanding these risks, ML may yet be applied to improve gain from imaging across all 3A's (access, acquisition, and assessment) for all patients. In this review, we present a framework for understanding the balance of opportunities and challenges for minimizing bias in medical imaging, how ML may improve current approaches to imaging, and what specific design considerations should be made as part of efforts to maximize the quality of health care for all.

The Unseen Hand: AI-Based Prescribing Decision Support Tools and the Evaluation of Drug Safety and Effectiveness

The use of artificial intelligence (AI)-based tools to guide prescribing decisions is full of promise and may enhance patient outcomes. These tools can perform actions such as choosing the 'safest' medication, choosing between competing medications, promoting de-prescribing or even predicting non-adherence. These tools can exist in a variety of formats; for example, they may be directly integrated into electronic medical records or they may exist in a stand-alone website accessible by a web browser. One potential impact of these tools is that they could manipulate our understanding of the benefit-risk of medicines in the real world. Currently, the benefit risk of approved medications is assessed according to carefully planned agreements covering spontaneous reporting systems and planned surveillance studies. But AI-based tools may limit or even block prescription to high-risk patients or prevent off-label use. The uptake and temporal availability of these tools may be uneven across healthcare systems and geographies, creating artefacts in data that are difficult to account for. It is also hard to estimate the 'true impact' that a tool had on a prescribing decision. International borders may also be highly porous to these tools, especially in cases where tools are available over the web. These tools already exist, and their use is likely to increase in the coming years. How they can be accounted for in benefit-risk decisions is yet to be seen.

Validating inborn error of immunity prevalence and risk with nationally representative electronic health record data

BACKGROUND

The 10 Warning Signs of Primary Immunodeficiency were created 30 years ago to advance recognition of inborn errors of immunity (IEI). However, no population-level assessment of their utility applied to electronic health record (EHR) data has been conducted.

OBJECTIVE

We sought to quantify the value of having ≥2 warning signs (WS) toward diagnosing IEI using a highly representative real-world US cohort. A secondary goal was estimating the US prevalence of IEI.

METHODS

In this cohort study, we accessed normalized and de-identified EHR data on 152 million US patients. An IEI cohort (n = 41,080), in which patients were defined by having at least 1 verifiable IEI diagnosis placed ≥2 times in their record, was compared with a matched set of controls (n = 250,262). WS were encoded along with relevant diagnoses, relative weights were calculated, and the proportion of IEI cases versus controls with ≥2 WS was compared.

RESULTS

The proportion of IEI cases with ≥2 WS significantly differed from controls (0.33 vs 0.031; P < .0005, χ2 test). We also estimated a US IEI prevalence of 6 per 10,000 individuals (41,080/73,165,655; 0.056%). WS 9 (≥2 deep-seated infections), 7 (fungal infections), 5 (failure to thrive) and 4 (≥2 pneumonias in 1 year) were the most heavily weighted among the IEI cohort.

CONCLUSIONS

This nationally representative US-based cohort study demonstrates that presence of WS and associated clinical diagnoses can facilitate identification of patients with IEI from EHR data. In addition, we estimate that 6 in 10,000, or approximately 150,000 to 200,000 individuals are affected by IEI across the United States.

A multinational study on the factors influencing university students' attitudes and usage of ChatGPT

Artificial intelligence models, like ChatGPT, have the potential to revolutionize higher education when implemented properly. This study aimed to investigate the factors influencing university students' attitudes and usage of ChatGPT in Arab countries. The survey instrument "TAME-ChatGPT" was administered to 2240 participants from Iraq, Kuwait, Egypt, Lebanon, and Jordan. Of those, 46.8% heard of ChatGPT, and 52.6% used it before the study. The results indicated that a positive attitude and usage of ChatGPT were determined by factors like ease of use, positive attitude towards technology, social influence, perceived usefulness, behavioral/cognitive influences, low perceived risks, and low anxiety. Confirmatory factor analysis indicated the adequacy of the "TAME-ChatGPT" constructs. Multivariate analysis demonstrated that the attitude towards ChatGPT usage was significantly influenced by country of residence, age, university type, and recent academic performance. This study validated "TAME-ChatGPT" as a useful tool for assessing ChatGPT adoption among university students. The successful integration of ChatGPT in higher education relies on the perceived ease of use, perceived usefulness, positive attitude towards technology, social influence, behavioral/cognitive elements, low anxiety, and minimal perceived risks. Policies for ChatGPT adoption in higher education should be tailored to individual contexts, considering the variations in student attitudes observed in this study.

Challenges of Teleneurology in the Care of Complex Neurodegenerative Disorders: The Case of Parkinson's Disease with Possible Solutions

Teleneurology is a specialist field within the realm of telemedicine, which is dedicated to delivering neurological care and consultations through virtual encounters. Teleneurology has been successfully used in acute care (e.g., stroke) and outpatient evaluation for chronic neurological conditions such as epilepsy and headaches. However, for some neurologic entities like Parkinson's disease, in which an in-depth physical examination by palpating muscles and performing neurologic maneuvers is the mainstay of monitoring the effects of medication, the yield and feasibility of a virtual encounter are low. Therefore, in this prospective review, we discuss two promising teleneurology approaches and propose adjustments to enhance the value of virtual encounters by improving the validity of neurological examination: 'hybrid teleneurology', which involves revising the workflow of virtual encounters; and 'artificial intelligence (AI)-assisted teleneurology', namely the use of biosensors and wearables and data processing using AI.

Risk of Bias in Chest Radiography Deep Learning Foundation Models

PURPOSE

To analyze a recently published chest radiography foundation model for the presence of biases that could lead to subgroup performance disparities across biologic sex and race.

MATERIALS AND METHODS

This Health Insurance Portability and Accountability Act-compliant retrospective study used 127 118 chest radiographs from 42 884 patients (mean age, 63 years ± 17 [SD]; 23 623 male, 19 261 female) from the CheXpert dataset that were collected between October 2002 and July 2017. To determine the presence of bias in features generated by a chest radiography foundation model and baseline deep learning model, dimensionality reduction methods together with two-sample Kolmogorov-Smirnov tests were used to detect distribution shifts across sex and race. A comprehensive disease detection performance analysis was then performed to associate any biases in the features to specific disparities in classification performance across patient subgroups.

RESULTS

Ten of 12 pairwise comparisons across biologic sex and race showed statistically significant differences in the studied foundation model, compared with four significant tests in the baseline model. Significant differences were found between male and female (P < .001) and Asian and Black (P < .001) patients in the feature projections that primarily capture disease. Compared with average model performance across all subgroups, classification performance on the "no finding" label decreased between 6.8% and 7.8% for female patients, and performance in detecting "pleural effusion" decreased between 10.7% and 11.6% for Black patients.

CONCLUSION

The studied chest radiography foundation model demonstrated racial and sex-related bias, which led to disparate performance across patient subgroups; thus, this model may be unsafe for clinical applications.Keywords: Conventional Radiography, Computer Application-Detection/Diagnosis, Chest Radiography, Bias, Foundation Models Supplemental material is available for this article. Published under a CC BY 4.0 license.See also commentary by Czum and Parr in this issue.

Variation in monitoring: Glucose measurement in the ICU as a case study to preempt spurious correlations

Objective

Health inequities can be influenced by demographic factors such as race and ethnicity, proficiency in English, and biological sex. Disparities may manifest as differential likelihood of testing which correlates directly with the likelihood of an intervention to address an abnormal finding. Our retrospective observational study evaluated the presence of variation in glucose measurements in the Intensive Care Unit (ICU).

Methods

Using the MIMIC-IV database (2008-2019), a single-center, academic referral hospital in Boston (USA), we identified adult patients meeting sepsis-3 criteria. Exclusion criteria were diabetic ketoacidosis, ICU length of stay under 1 day, and unknown race or ethnicity. We performed a logistic regression analysis to assess differential likelihoods of glucose measurements on day 1. A negative binomial regression was fitted to assess the frequency of subsequent glucose readings. Analyses were adjusted for relevant clinical confounders, and performed across three disparity proxy axes: race and ethnicity, sex, and English proficiency.

Results

We studied 24,927 patients, of which 19.5% represented racial and ethnic minority groups, 42.4% were female, and 9.8% had limited English proficiency. No significant differences were found for glucose measurement on day 1 in the ICU. This pattern was consistent irrespective of the axis of analysis, i.e. race and ethnicity, sex, or English proficiency. Conversely, subsequent measurement frequency revealed potential disparities. Specifically, males (incidence rate ratio (IRR) 1.06, 95% confidence interval (CI) 1.01 - 1.21), patients who identify themselves as Hispanic (IRR 1.11, 95% CI 1.01 - 1.21), or Black (IRR 1.06, 95% CI 1.01 - 1.12), and patients being English proficient (IRR 1.08, 95% CI 1.01 - 1.15) had higher chances of subsequent glucose readings.

Conclusion

We found disparities in ICU glucose measurements among patients with sepsis, albeit the magnitude was small. Variation in disease monitoring is a source of data bias that may lead to spurious correlations when modeling health data.

Strategies and solutions to address Digital Determinants of Health (DDOH) across underinvested communities

Healthcare has long struggled to improve services through technology without further widening health disparities. With the significant expansion of digital health, a group of healthcare professionals and scholars from across the globe are proposing the official usage of the term "Digital Determinants of Health" (DDOH) to explicitly call out the relationship between technology, healthcare, and equity. This is the final paper in a series published in PLOS Digital Health that seeks to understand and summarize current knowledge of the strategies and solutions that help to mitigate the negative effects of DDOH for underinvested communities. Through a search of English-language Medline, Scopus, and Google Scholar articles published since 2010, 345 articles were identified that discussed the application of digital health technology among underinvested communities. A group of 8 reviewers assessed 132 articles selected at random for the mention of solutions that minimize differences in DDOH. Solutions were then organized by categories of policy; design and development; implementation and adoption; and evaluation and ongoing monitoring. The data were then assessed by category and the findings summarized. The reviewers also looked for common themes across the solutions and evidence of effectiveness. From this limited scoping review, the authors found numerous solutions mentioned across the papers for addressing DDOH and many common themes emerged regardless of the specific community or digital health technology under review. There was notably less information on solutions regarding ongoing evaluation and monitoring which corresponded with a lack of research evidence regarding effectiveness. The findings directionally suggest that universal strategies and solutions can be developed to address DDOH independent of the specific community under focus. With the need for the further development of DDOH measures, we also provide a framework for DDOH assessment.