Addressing health disparities in the Food and Drug Administration's artificial intelligence and machine learning regulatory framework

Artificial Intelligence in Endocrinology: On Track Toward Great Opportunities

In endocrinology, the types and quantity of digital data are increasing rapidly. Computing capabilities are also developing at an incredible rate, as illustrated by the recent expansion in the use of popular generative artificial intelligence (AI) applications. Numerous diagnostic and therapeutic devices using AI have already entered routine endocrine practice, and developments in this field are expected to continue to accelerate. Endocrinologists will need to be supported in managing AI applications. Beyond technological training, interdisciplinary vision is needed to encompass the ethical and legal aspects of AI, to manage the profound impact of AI on patient/provider relationships, and to maintain an optimal balance between human input and AI in endocrinology.

Evaluating Algorithmic Bias in 30-Day Hospital Readmission Models: Retrospective Analysis

BACKGROUND

The adoption of predictive algorithms in health care comes with the potential for algorithmic bias, which could exacerbate existing disparities. Fairness metrics have been proposed to measure algorithmic bias, but their application to real-world tasks is limited.

OBJECTIVE

This study aims to evaluate the algorithmic bias associated with the application of common 30-day hospital readmission models and assess the usefulness and interpretability of selected fairness metrics.

METHODS

We used 10.6 million adult inpatient discharges from Maryland and Florida from 2016 to 2019 in this retrospective study. Models predicting 30-day hospital readmissions were evaluated: LACE Index, modified HOSPITAL score, and modified Centers for Medicare & Medicaid Services (CMS) readmission measure, which were applied as-is (using existing coefficients) and retrained (recalibrated with 50% of the data). Predictive performances and bias measures were evaluated for all, between Black and White populations, and between low- and other-income groups. Bias measures included the parity of false negative rate (FNR), false positive rate (FPR), 0-1 loss, and generalized entropy index. Racial bias represented by FNR and FPR differences was stratified to explore shifts in algorithmic bias in different populations.

RESULTS

The retrained CMS model demonstrated the best predictive performance (area under the curve: 0.74 in Maryland and 0.68-0.70 in Florida), and the modified HOSPITAL score demonstrated the best calibration (Brier score: 0.16-0.19 in Maryland and 0.19-0.21 in Florida). Calibration was better in White (compared to Black) populations and other-income (compared to low-income) groups, and the area under the curve was higher or similar in the Black (compared to White) populations. The retrained CMS and modified HOSPITAL score had the lowest racial and income bias in Maryland. In Florida, both of these models overall had the lowest income bias and the modified HOSPITAL score showed the lowest racial bias. In both states, the White and higher-income populations showed a higher FNR, while the Black and low-income populations resulted in a higher FPR and a higher 0-1 loss. When stratified by hospital and population composition, these models demonstrated heterogeneous algorithmic bias in different contexts and populations.

CONCLUSIONS

Caution must be taken when interpreting fairness measures' face value. A higher FNR or FPR could potentially reflect missed opportunities or wasted resources, but these measures could also reflect health care use patterns and gaps in care. Simply relying on the statistical notions of bias could obscure or underplay the causes of health disparity. The imperfect health data, analytic frameworks, and the underlying health systems must be carefully considered. Fairness measures can serve as a useful routine assessment to detect disparate model performances but are insufficient to inform mechanisms or policy changes. However, such an assessment is an important first step toward data-driven improvement to address existing health disparities.

Ethnic disparity in diagnosing asymptomatic bacterial vaginosis using machine learning

While machine learning (ML) has shown great promise in medical diagnostics, a major challenge is that ML models do not always perform equally well among ethnic groups. This is alarming for women's health, as there are already existing health disparities that vary by ethnicity. Bacterial Vaginosis (BV) is a common vaginal syndrome among women of reproductive age and has clear diagnostic differences among ethnic groups. Here, we investigate the ability of four ML algorithms to diagnose BV. We determine the fairness in the prediction of asymptomatic BV using 16S rRNA sequencing data from Asian, Black, Hispanic, and white women. General purpose ML model performances vary based on ethnicity. When evaluating the metric of false positive or false negative rate, we find that models perform least effectively for Hispanic and Asian women. Models generally have the highest performance for white women and the lowest for Asian women. These findings demonstrate a need for improved methodologies to increase model fairness for predicting BV.

A normative framework for artificial intelligence as a sociotechnical system in healthcare

Artificial intelligence (AI) tools are of great interest to healthcare organizations for their potential to improve patient care, yet their translation into clinical settings remains inconsistent. One of the reasons for this gap is that good technical performance does not inevitably result in patient benefit. We advocate for a conceptual shift wherein AI tools are seen as components of an intervention ensemble. The intervention ensemble describes the constellation of practices that, together, bring about benefit to patients or health systems. Shifting from a narrow focus on the tool itself toward the intervention ensemble prioritizes a "sociotechnical" vision for translation of AI that values all components of use that support beneficial patient outcomes. The intervention ensemble approach can be used for regulation, institutional oversight, and for AI adopters to responsibly and ethically appraise, evaluate, and use AI tools.

The value of standards for health datasets in artificial intelligence-based applications

Artificial intelligence as a medical device is increasingly being applied to healthcare for diagnosis, risk stratification and resource allocation. However, a growing body of evidence has highlighted the risk of algorithmic bias, which may perpetuate existing health inequity. This problem arises in part because of systemic inequalities in dataset curation, unequal opportunity to participate in research and inequalities of access. This study aims to explore existing standards, frameworks and best practices for ensuring adequate data diversity in health datasets. Exploring the body of existing literature and expert views is an important step towards the development of consensus-based guidelines. The study comprises two parts: a systematic review of existing standards, frameworks and best practices for healthcare datasets; and a survey and thematic analysis of stakeholder views of bias, health equity and best practices for artificial intelligence as a medical device. We found that the need for dataset diversity was well described in literature, and experts generally favored the development of a robust set of guidelines, but there were mixed views about how these could be implemented practically. The outputs of this study will be used to inform the development of standards for transparency of data diversity in health datasets (the STANDING Together initiative).

Fairness as an afterthought: An American perspective on fairness in model developer-clinician user collaborations

Numerous ethics guidelines have been handed down over the last few years on the ethical applications of machine learning models. Virtually every one of them mentions the importance of "fairness" in the development and use of these models. Unfortunately, though, these ethics documents omit providing a consensually adopted definition or characterization of fairness. As one group of authors observed, these documents treat fairness as an "afterthought" whose importance is undeniable but whose essence seems strikingly elusive. In this essay, which offers a distinctly American treatment of "fairness," we comment on a number of fairness formulations and on qualitative or statistical methods that have been encouraged to achieve fairness. We argue that none of them, at least from an American moral perspective, provides a one-size-fits-all definition of or methodology for securing fairness that could inform or standardize fairness over the universe of use cases witnessing machine learning applications. Instead, we argue that because fairness comprehensions and applications reflect a vast range of use contexts, model developers and clinician users will need to engage in thoughtful collaborations that examine how fairness should be conceived and operationalized in the use case at issue. Part II of this paper illustrates key moments in these collaborations, especially when inter and intra disagreement occurs among model developer and clinician user groups over whether a model is fair or unfair. We conclude by noting that these collaborations will likely occur over the lifetime of a model if its claim to fairness is to advance beyond "afterthought" status.

Research agenda for using artificial intelligence in health governance: interpretive scoping review and framework

BACKGROUND

The governance of health systems is complex in nature due to several intertwined and multi-dimensional factors contributing to it. Recent challenges of health systems reflect the need for innovative approaches that can minimize adverse consequences of policies. Hence, there is compelling evidence of a distinct outlook on the health ecosystem using artificial intelligence (AI). Therefore, this study aimed to investigate the roles of AI and its applications in health system governance through an interpretive scoping review of current evidence.

METHOD

This study intended to offer a research agenda and framework for the applications of AI in health systems governance. To include shreds of evidence with a greater focus on the application of AI in health governance from different perspectives, we searched the published literature from 2000 to 2023 through PubMed, Scopus, and Web of Science Databases.

RESULTS

Our findings showed that integrating AI capabilities into health systems governance has the potential to influence three cardinal dimensions of health. These include social determinants of health, elements of governance, and health system tasks and goals. AI paves the way for strengthening the health system's governance through various aspects, i.e., intelligence innovations, flexible boundaries, multidimensional analysis, new insights, and cognition modifications to the health ecosystem area.

CONCLUSION

AI is expected to be seen as a tool with new applications and capabilities, with the potential to change each component of governance in the health ecosystem, which can eventually help achieve health-related goals.

Applying anti-racist approaches to informatics: a new lens on traditional frames

Health organizations and systems rely on increasingly sophisticated informatics infrastructure. Without anti-racist expertise, the field risks reifying and entrenching racism in information systems. We consider ways the informatics field can recognize institutional, systemic, and structural racism and propose the use of the Public Health Critical Race Praxis (PHCRP) to mitigate and dismantle racism in digital forms. We enumerate guiding questions for stakeholders along with a PHCRP-Informatics framework. By focusing on (1) critical self-reflection, (2) following the expertise of well-established scholars of racism, (3) centering the voices of affected individuals and communities, and (4) critically evaluating practice resulting from informatics systems, stakeholders can work to minimize the impacts of racism. Informatics, informed and guided by this proposed framework, will help realize the vision of health systems that are more fair, just, and equitable.

Imaging Artificial Intelligence: A Framework for Radiologists to Address Health Equity, From the AJR Special Series on DEI

Artificial intelligence (AI) holds promise for helping patients access new and individualized health care pathways while increasing efficiencies for health care practitioners. Radiology has been at the forefront of this technology in medicine; many radiology practices are implementing and trialing AI-focused products. AI also holds great promise for reducing health disparities and promoting health equity. Radiology is ideally positioned to help reduce disparities given its central and critical role in patient care. The purposes of this article are to discuss the potential benefits and pitfalls of deploying AI algorithms in radiology, specifically highlighting the impact of AI on health equity; to explore ways to mitigate drivers of inequity; and to enhance pathways for creating better health care for all individuals, centering on a practical framework that helps radiologists address health equity during deployment of new tools.

Evaluating equity in performance of an electronic health record-based 6-month mortality risk model to trigger palliative care consultation: a retrospective model validation analysis

OBJECTIVE

Evaluate predictive performance of an electronic health record (EHR)-based, inpatient 6-month mortality risk model developed to trigger palliative care consultation among patient groups stratified by age, race, ethnicity, insurance and socioeconomic status (SES), which may vary due to social forces (eg, racism) that shape health, healthcare and health data.

DESIGN

Retrospective evaluation of prediction model.

SETTING

Three urban hospitals within a single health system.

PARTICIPANTS

All patients ≥18 years admitted between 1 January and 31 December 2017, excluding observation, obstetric, rehabilitation and hospice (n=58 464 encounters, 41 327 patients).

MAIN OUTCOME MEASURES

General performance metrics (c-statistic, integrated calibration index (ICI), Brier Score) and additional measures relevant to health equity (accuracy, false positive rate (FPR), false negative rate (FNR)).

RESULTS

For black versus non-Hispanic white patients, the model's accuracy was higher (0.051, 95% CI 0.044 to 0.059), FPR lower (-0.060, 95% CI -0.067 to -0.052) and FNR higher (0.049, 95% CI 0.023 to 0.078). A similar pattern was observed among patients who were Hispanic, younger, with Medicaid/missing insurance, or living in low SES zip codes. No consistent differences emerged in c-statistic, ICI or Brier Score. Younger age had the second-largest effect size in the mortality prediction model, and there were large standardised group differences in age (eg, 0.32 for non-Hispanic white versus black patients), suggesting age may contribute to systematic differences in the predicted probabilities between groups.

CONCLUSIONS

An EHR-based mortality risk model was less likely to identify some marginalised patients as potentially benefiting from palliative care, with younger age pinpointed as a possible mechanism. Evaluating predictive performance is a critical preliminary step in addressing algorithmic inequities in healthcare, which must also include evaluating clinical impact, and governance and regulatory structures for oversight, monitoring and accountability.

Epistemic Rights and Responsibilities of Digital Simulacra for Biomedicine

Big data and AI have enabled digital simulation for prediction of future health states or behaviors of specific individuals, populations or humans in general. "Digital simulacra" use multimodal datasets to develop computational models that are virtual representations of people or groups, generating predictions of how systems evolve and react to interventions over time. These include digital twins and virtual patients for in silico clinical trials, both of which seek to transform research and health care by speeding innovation and bridging the epistemic gap between population-based research findings and their application to the individual. Nevertheless, digital simulacra mark a major milestone on a trajectory to embrace the epistemic culture of data science and a potential abandonment of medical epistemological concepts of causality and representation. In doing so, "data first" approaches potentially shift moral attention from actual patients and principles, such as equity, to simulated patients and patient data.

From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment

Machine learning (ML) is increasingly used in clinical oncology to diagnose cancers, predict patient outcomes, and inform treatment planning. Here, we review recent applications of ML across the clinical oncology workflow. We review how these techniques are applied to medical imaging and to molecular data obtained from liquid and solid tumor biopsies for cancer diagnosis, prognosis, and treatment design. We discuss key considerations in developing ML for the distinct challenges posed by imaging and molecular data. Finally, we examine ML models approved for cancer-related patient usage by regulatory agencies and discuss approaches to improve the clinical usefulness of ML.

Leveraging Clinical Informatics and Data Science to Improve Care and Facilitate Research in Pediatric Acute Respiratory Distress Syndrome: From the Second Pediatric Acute Lung Injury Consensus Conference

OBJECTIVES

The use of electronic algorithms, clinical decision support systems, and other clinical informatics interventions is increasing in critical care. Pediatric acute respiratory distress syndrome (PARDS) is a complex, dynamic condition associated with large amounts of clinical data and frequent decisions at the bedside. Novel data-driven technologies that can help screen, prompt, and support clinician decision-making could have a significant impact on patient outcomes. We sought to identify and summarize relevant evidence related to clinical informatics interventions in both PARDS and adult respiratory distress syndrome (ARDS), for the second Pediatric Acute Lung Injury Consensus Conference.

DATA SOURCES

MEDLINE (Ovid), Embase (Elsevier), and CINAHL Complete (EBSCOhost).

STUDY SELECTION

We included studies of pediatric or adult critically ill patients with or at risk of ARDS that examined automated screening tools, electronic algorithms, or clinical decision support systems.

DATA EXTRACTION

Title/abstract review, full text review, and data extraction using a standardized data extraction form.

DATA SYNTHESIS

The Grading of Recommendations Assessment, Development and Evaluation approach was used to identify and summarize evidence and develop recommendations. Twenty-six studies were identified for full text extraction to address the Patient/Intervention/Comparator/Outcome questions, and 14 were used for the recommendations/statements. Two clinical recommendations were generated, related to the use of electronic screening tools and automated monitoring of compliance with best practice guidelines. Two research statements were generated, related to the development of multicenter data collaborations and the design of generalizable algorithms and electronic tools. One policy statement was generated, related to the provision of material and human resources by healthcare organizations to empower clinicians to develop clinical informatics interventions to improve the care of patients with PARDS.

CONCLUSIONS

We present two clinical recommendations and three statements (two research one policy) for the use of electronic algorithms and clinical informatics tools for patients with PARDS based on a systematic review of the literature and expert consensus.

Artificial intelligence and health inequities in primary care: a systematic scoping review and framework

OBJECTIVE

Artificial intelligence (AI) will have a significant impact on healthcare over the coming decade. At the same time, health inequity remains one of the biggest challenges. Primary care is both a driver and a mitigator of health inequities and with AI gaining traction in primary care, there is a need for a holistic understanding of how AI affect health inequities, through the act of providing care and through potential system effects. This paper presents a systematic scoping review of the ways AI implementation in primary care may impact health inequity.

DESIGN

Following a systematic scoping review approach, we searched for literature related to AI, health inequity, and implementation challenges of AI in primary care. In addition, articles from primary exploratory searches were added, and through reference screening.The results were thematically summarised and used to produce both a narrative and conceptual model for the mechanisms by which social determinants of health and AI in primary care could interact to either improve or worsen health inequities.Two public advisors were involved in the review process.

ELIGIBILITY CRITERIA

Peer-reviewed publications and grey literature in English and Scandinavian languages.

INFORMATION SOURCES

PubMed, SCOPUS and JSTOR.

RESULTS

A total of 1529 publications were identified, of which 86 met the inclusion criteria. The findings were summarised under six different domains, covering both positive and negative effects: (1) access, (2) trust, (3) dehumanisation, (4) agency for self-care, (5) algorithmic bias and (6) external effects. The five first domains cover aspects of the interface between the patient and the primary care system, while the last domain covers care system-wide and societal effects of AI in primary care. A graphical model has been produced to illustrate this. Community involvement throughout the whole process of designing and implementing of AI in primary care was a common suggestion to mitigate the potential negative effects of AI.

CONCLUSION

AI has the potential to affect health inequities through a multitude of ways, both directly in the patient consultation and through transformative system effects. This review summarises these effects from a system tive and provides a base for future research into responsible implementation.

Radiomics in neuro-oncological clinical trials

The development of clinical trials has led to substantial improvements in the prevention and treatment of many diseases, including brain cancer. Advances in medicine, such as improved surgical techniques, the development of new drugs and devices, the use of statistical methods in research, and the development of codes of ethics, have considerably influenced the way clinical trials are conducted today. In addition, methods from the broad field of artificial intelligence, such as radiomics, have the potential to considerably affect clinical trials and clinical practice in the future. Radiomics is a method to extract undiscovered features from routinely acquired imaging data that can neither be captured by means of human perception nor conventional image analysis. In patients with brain cancer, radiomics has shown its potential for the non-invasive identification of prognostic biomarkers, automated response assessment, and differentiation between treatment-related changes from tumour progression. Despite promising results, radiomics is not yet established in routine clinical practice nor in clinical trials. In this Viewpoint, the European Organization for Research and Treatment of Cancer Brain Tumour Group summarises the current status of radiomics, discusses its potential and limitations, envisions its future role in clinical trials in neuro-oncology, and provides guidance on how to address the challenges in radiomics.

Systematic analysis of the test design and performance of AI/ML-based medical devices approved for triage/detection/diagnosis in the USA and Japan

The development of computer-aided detection (CAD) using artificial intelligence (AI) and machine learning (ML) is rapidly evolving. Submission of AI/ML-based CAD devices for regulatory approval requires information about clinical trial design and performance criteria, but the requirements vary between countries. This study compares the requirements for AI/ML-based CAD devices approved by the US Food and Drug Administration (FDA) and the Pharmaceuticals and Medical Devices Agency (PMDA) in Japan. A list of 45 FDA-approved and 12 PMDA-approved AI/ML-based CAD devices was compiled. In the USA, devices classified as computer-aided simple triage were approved based on standalone software testing, whereas devices classified as computer-aided detection/diagnosis were approved based on reader study testing. In Japan, however, there was no clear distinction between evaluation methods according to the category. In the USA, a prospective randomized controlled trial was conducted for AI/ML-based CAD devices used for the detection of colorectal polyps, whereas in Japan, such devices were approved based on standalone software testing. This study indicated that the different viewpoints of AI/ML-based CAD in the two countries influenced the selection of different evaluation methods. This study’s findings may be useful for defining a unified global development and approval standard for AI/ML-based CAD.

Challenges in translational machine learning

Machine learning (ML) algorithms are increasingly being used to help implement clinical decision support systems. In this new field, we define as "translational machine learning", joint efforts and strong communication between data scientists and clinicians help to span the gap between ML and its adoption in the clinic. These collaborations also improve interpretability and trust in translational ML methods and ultimately aim to result in generalizable and reproducible models. To help clinicians and bioinformaticians refine their translational ML pipelines, we review the steps from model building to the use of ML in the clinic. We discuss experimental setup, computational analysis, interpretability and reproducibility, and emphasize the challenges involved. We highly advise collaboration and data sharing between consortia and institutes to build multi-centric cohorts that facilitate ML methodologies that generalize across centers. In the end, we hope that this review provides a way to streamline translational ML and helps to tackle the challenges that come with it.

Assessing socioeconomic bias in machine learning algorithms in health care: a case study of the HOUSES index

OBJECTIVE

Artificial intelligence (AI) models may propagate harmful biases in performance and hence negatively affect the underserved. We aimed to assess the degree to which data quality of electronic health records (EHRs) affected by inequities related to low socioeconomic status (SES), results in differential performance of AI models across SES.

MATERIALS AND METHODS

This study utilized existing machine learning models for predicting asthma exacerbation in children with asthma. We compared balanced error rate (BER) against different SES levels measured by HOUsing-based SocioEconomic Status measure (HOUSES) index. As a possible mechanism for differential performance, we also compared incompleteness of EHR information relevant to asthma care by SES.

RESULTS

Asthmatic children with lower SES had larger BER than those with higher SES (eg, ratio = 1.35 for HOUSES Q1 vs Q2-Q4) and had a higher proportion of missing information relevant to asthma care (eg, 41% vs 24% for missing asthma severity and 12% vs 9.8% for undiagnosed asthma despite meeting asthma criteria).

DISCUSSION

Our study suggests that lower SES is associated with worse predictive model performance. It also highlights the potential role of incomplete EHR data in this differential performance and suggests a way to mitigate this bias.

CONCLUSION

The HOUSES index allows AI researchers to assess bias in predictive model performance by SES. Although our case study was based on a small sample size and a single-site study, the study results highlight a potential strategy for identifying bias by using an innovative SES measure.

Discrimination, trust, and withholding information from providers: Implications for missing data and inequity

Quality care requires collaborative communication, information exchange, and decision-making between patients and providers. Complete and accurate data about patients and from patients are especially important as high volumes of data are used to build clinical decision support tools and inform precision medicine initiatives. However, systematically missing data can bias these tools and threaten their effectiveness. Data completeness relies in many ways on patients being comfortable disclosing information to their providers without prohibitive concerns about security or privacy. Patients are likely to withhold information in the context of low trust relationships with providers, but it is unknown how experiences of discrimination in the healthcare system also relate to non-disclosure. In this study, we assess the relationship between withholding information from providers, experiences of discrimination, and multiple types of patient trust. Using a nationally representative sample of US adults (n = 2,029), weighted logistic regression modeling indicated a statistically significant relationship between experiences of discrimination and withholding information from providers (OR 3.7; CI [2.6-5.2], p < .001). Low trust in provider disclosure of conflicts of interest and low trust in providers' responsible use of health information were also positively associated with non-disclosure. We further analyzed the relationship between non-disclosure and the five most common types of discrimination (e.g., discrimination based on race, education/income, weight, gender, and age). We observed that all five types were statistically significantly associated with non-disclosure (p < .05). These results suggest that experiences of discrimination and specific types of low trust have a meaningful association with a patient's willingness to share information with their provider, with important implications for the quality of data available for medical decision-making and care. Because incomplete information can contribute to lower quality care, especially in the context of data-driven decision-making, patients experiencing discrimination may be further disadvantaged and harmed by systematic data missingness in their records.

A Research Ethics Framework for the Clinical Translation of Healthcare Machine Learning

The application of artificial intelligence and machine learning (ML) technologies in healthcare have immense potential to improve the care of patients. While there are some emerging practices surrounding responsible ML as well as regulatory frameworks, the traditional role of research ethics oversight has been relatively unexplored regarding its relevance for clinical ML. In this paper, we provide a comprehensive research ethics framework that can apply to the systematic inquiry of ML research across its development cycle. The pathway consists of three stages: (1) exploratory, hypothesis-generating data access; (2) silent period evaluation; (3) prospective clinical evaluation. We connect each stage to its literature and ethical justification and suggest adaptations to traditional paradigms to suit ML while maintaining ethical rigor and the protection of individuals. This pathway can accommodate a multitude of research designs from observational to controlled trials, and the stages can apply individually to a variety of ML applications.

Ethical, legal, and social considerations of AI-based medical decision-support tools: A scoping review

INTRODUCTION

Recent developments in the field of Artificial Intelligence (AI) applied to healthcare promise to solve many of the existing global issues in advancing human health and managing global health challenges. This comprehensive review aims not only to surface the underlying ethical and legal but also social implications (ELSI) that have been overlooked in recent reviews while deserving equal attention in the development stage, and certainly ahead of implementation in healthcare. It is intended to guide various stakeholders (eg. designers, engineers, clinicians) in addressing the ELSI of AI at the design stage using the Ethics by Design (EbD) approach.

METHODS

The authors followed a systematised scoping methodology and searched the following databases: Pubmed, Web of science, Ovid, Scopus, IEEE Xplore, EBSCO Search (Academic Search Premier, CINAHL, PSYCINFO, APA PsycArticles, ERIC) for the ELSI of AI in healthcare through January 2021. Data were charted and synthesised, and the authors conducted a descriptive and thematic analysis of the collected data.

RESULTS

After reviewing 1108 papers, 94 were included in the final analysis. Our results show a growing interest in the academic community for ELSI in the field of AI. The main issues of concern identified in our analysis fall into four main clusters of impact: AI algorithms, physicians, patients, and healthcare in general. The most prevalent issues are patient safety, algorithmic transparency, lack of proper regulation, liability & accountability, impact on patient-physician relationship and governance of AI empowered healthcare.

CONCLUSIONS

The results of our review confirm the potential of AI to significantly improve patient care, but the drawbacks to its implementation relate to complex ELSI that have yet to be addressed. Most ELSI refer to the impact on and extension of the reciprocal and fiduciary patient-physician relationship. With the integration of AIbased decision making tools, a bilateral patient-physician relationship may shift into a trilateral one.

Ethical Machine Learning in Healthcare

The use of machine learning (ML) in healthcare raises numerous ethical concerns, especially as models can amplify existing health inequities. Here, we outline ethical considerations for equitable ML in the advancement of healthcare. Specifically, we frame ethics of ML in healthcare through the lens of social justice. We describe ongoing efforts and outline challenges in a proposed pipeline of ethical ML in health, ranging from problem selection to postdeployment considerations. We close by summarizing recommendations to address these challenges.

Ensuring that biomedical AI benefits diverse populations

Artificial Intelligence (AI) can potentially impact many aspects of human health, from basic research discovery to individual health assessment. It is critical that these advances in technology broadly benefit diverse populations from around the world. This can be challenging because AI algorithms are often developed on non-representative samples and evaluated based on narrow metrics. Here we outline key challenges to biomedical AI in outcome design, data collection and technology evaluation, and use examples from precision health to illustrate how bias and health disparity may arise in each stage. We then suggest both short term approaches-more diverse data collection and AI monitoring-and longer term structural changes in funding, publications, and education to address these challenges.

Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper

The development and implementation of clinical decision support (CDS) that trains itself and adapts its algorithms based on new data-here referred to as Adaptive CDS-present unique challenges and considerations. Although Adaptive CDS represents an expected progression from earlier work, the activities needed to appropriately manage and support the establishment and evolution of Adaptive CDS require new, coordinated initiatives and oversight that do not currently exist. In this AMIA position paper, the authors describe current and emerging challenges to the safe use of Adaptive CDS and lay out recommendations for the effective management and monitoring of Adaptive CDS.