Identifying Ethical Considerations for Machine Learning Healthcare Applications

Reimagining Healthcare: Unleashing the Power of Artificial Intelligence in Medicine

Artificial intelligence (AI) has opened new medical avenues and revolutionized diagnostic and therapeutic practices, allowing healthcare providers to overcome significant challenges associated with cost, disease management, accessibility, and treatment optimization. Prominent AI technologies such as machine learning (ML) and deep learning (DL) have immensely influenced diagnostics, patient monitoring, novel pharmaceutical discoveries, drug development, and telemedicine. Significant innovations and improvements in disease identification and early intervention have been made using AI-generated algorithms for clinical decision support systems and disease prediction models. AI has remarkably impacted clinical drug trials by amplifying research into drug efficacy, adverse events, and candidate molecular design. AI's precision and analysis regarding patients' genetic, environmental, and lifestyle factors have led to individualized treatment strategies. During the COVID-19 pandemic, AI-assisted telemedicine set a precedent for remote healthcare delivery and patient follow-up. Moreover, AI-generated applications and wearable devices have allowed ambulatory monitoring of vital signs. However, apart from being immensely transformative, AI's contribution to healthcare is subject to ethical and regulatory concerns. AI-backed data protection and algorithm transparency should be strictly adherent to ethical principles. Vigorous governance frameworks should be in place before incorporating AI in mental health interventions through AI-operated chatbots, medical education enhancements, and virtual reality-based training. The role of AI in medical decision-making has certain limitations, necessitating the importance of hands-on experience. Therefore, reaching an optimal balance between AI's capabilities and ethical considerations to ensure impartial and neutral performance in healthcare applications is crucial. This narrative review focuses on AI's impact on healthcare and the importance of ethical and balanced incorporation to make use of its full potential.

Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework

OBJECTIVE

To derive a comprehensive implementation framework for clinical AI models within hospitals informed by existing AI frameworks and integrated with reporting standards for clinical AI research.

MATERIALS AND METHODS

(1) Derive a provisional implementation framework based on the taxonomy of Stead et al and integrated with current reporting standards for AI research: TRIPOD, DECIDE-AI, CONSORT-AI. (2) Undertake a scoping review of published clinical AI implementation frameworks and identify key themes and stages. (3) Perform a gap analysis and refine the framework by incorporating missing items.

RESULTS

The provisional AI implementation framework, called SALIENT, was mapped to 5 stages common to both the taxonomy and the reporting standards. A scoping review retrieved 20 studies and 247 themes, stages, and subelements were identified. A gap analysis identified 5 new cross-stage themes and 16 new tasks. The final framework comprised 5 stages, 7 elements, and 4 components, including the AI system, data pipeline, human-computer interface, and clinical workflow.

DISCUSSION

This pragmatic framework resolves gaps in existing stage- and theme-based clinical AI implementation guidance by comprehensively addressing the what (components), when (stages), and how (tasks) of AI implementation, as well as the who (organization) and why (policy domains). By integrating research reporting standards into SALIENT, the framework is grounded in rigorous evaluation methodologies. The framework requires validation as being applicable to real-world studies of deployed AI models.

CONCLUSIONS

A novel end-to-end framework has been developed for implementing AI within hospital clinical practice that builds on previous AI implementation frameworks and research reporting standards.

Advancing AI in healthcare: A comprehensive review of best practices

Artificial Intelligence (AI) and Machine Learning (ML) are powerful tools shaping the healthcare sector. This review considers twelve key aspects of AI in clinical practice: 1) Ethical AI; 2) Explainable AI; 3) Health Equity and Bias in AI; 4) Sponsorship Bias; 5) Data Privacy; 6) Genomics and Privacy; 7) Insufficient Sample Size and Self-Serving Bias; 8) Bridging the Gap Between Training Datasets and Real-World Scenarios; 9) Open Source and Collaborative Development; 10) Dataset Bias and Synthetic Data; 11) Measurement Bias; 12) Reproducibility in AI Research. These categories represent both the challenges and opportunities of AI implementation in healthcare. While AI holds significant potential for improving patient care, it also presents risks and challenges, such as ensuring privacy, combating bias, and maintaining transparency and ethics. The review underscores the necessity of developing comprehensive best practices for healthcare organizations and fostering a diverse dialogue involving data scientists, clinicians, patient advocates, ethicists, economists, and policymakers. We are at the precipice of significant transformation in healthcare powered by AI. By continuing to reassess and refine our approach, we can ensure that AI is implemented responsibly and ethically, maximizing its benefit to patient care and public health.

Towards artificial intelligence-enabled extracellular vesicle precision drug delivery

Extracellular Vesicles (EVs), particularly exosomes, recently exploded into nanomedicine as an emerging drug delivery approach due to their superior biocompatibility, circulating stability, and bioavailability in vivo. However, EV heterogeneity makes molecular targeting precision a critical challenge. Deciphering key molecular drivers for controlling EV tissue targeting specificity is in great need. Artificial intelligence (AI) brings powerful prediction ability for guiding the rational design of engineered EVs in precision control for drug delivery. This review focuses on cutting-edge nano-delivery via integrating large-scale EV data with AI to develop AI-directed EV therapies and illuminate the clinical translation potential. We briefly review the current status of EVs in drug delivery, including the current frontier, limitations, and considerations to advance the field. Subsequently, we detail the future of AI in drug delivery and its impact on precision EV delivery. Our review discusses the current universal challenge of standardization and critical considerations when using AI combined with EVs for precision drug delivery. Finally, we will conclude this review with a perspective on future clinical translation led by a combined effort of AI and EV research.

A Methodology for a Scalable, Collaborative, and Resource-Efficient Platform, MERLIN, to Facilitate Healthcare AI Research

Healthcare artificial intelligence (AI) holds the potential to increase patient safety, augment efficiency and improve patient outcomes, yet research is often limited by data access, cohort curation, and tools for analysis. Collection and translation of electronic health record data, live data, and real-time high-resolution device data can be challenging and time-consuming. The development of clinically relevant AI tools requires overcoming challenges in data acquisition, scarce hospital resources, and requirements for data governance. These bottlenecks may result in resource-heavy needs and long delays in research and development of AI systems. We present a system and methodology to accelerate data acquisition, dataset development and analysis, and AI model development. We created an interactive platform that relies on a scalable microservice architecture. This system can ingest 15,000 patient records per hour, where each record represents thousands of multimodal measurements, text notes, and high-resolution data. Collectively, these records can approach a terabyte of data. The platform can further perform cohort generation and preliminary dataset analysis in 2-5 minutes. As a result, multiple users can collaborate simultaneously to iterate on datasets and models in real time. We anticipate that this approach will accelerate clinical AI model development, and, in the long run, meaningfully improve healthcare delivery.

Reflections on Putting AI Ethics into Practice: How Three AI Ethics Approaches Conceptualize Theory and Practice

Critics currently argue that applied ethics approaches to artificial intelligence (AI) are too principles-oriented and entail a theory-practice gap. Several applied ethical approaches try to prevent such a gap by conceptually translating ethical theory into practice. In this article, we explore how the currently most prominent approaches of AI ethics translate ethics into practice. Therefore, we examine three approaches to applied AI ethics: the embedded ethics approach, the ethically aligned approach, and the Value Sensitive Design (VSD) approach. We analyze each of these three approaches by asking how they understand and conceptualize theory and practice. We outline the conceptual strengths as well as their shortcomings: an embedded ethics approach is context-oriented but risks being biased by it; ethically aligned approaches are principles-oriented but lack justification theories to deal with trade-offs between competing principles; and the interdisciplinary Value Sensitive Design approach is based on stakeholder values but needs linkage to political, legal, or social governance aspects. Against this background, we develop a meta-framework for applied AI ethics conceptions with three dimensions. Based on critical theory, we suggest these dimensions as starting points to critically reflect on the conceptualization of theory and practice. We claim, first, that the inclusion of the dimension of affects and emotions in the ethical decision-making process stimulates reflections on vulnerabilities, experiences of disregard, and marginalization already within the AI development process. Second, we derive from our analysis that considering the dimension of justifying normative background theories provides both standards and criteria as well as guidance for prioritizing or evaluating competing principles in cases of conflict. Third, we argue that reflecting the governance dimension in ethical decision-making is an important factor to reveal power structures as well as to realize ethical AI and its application because this dimension seeks to combine social, legal, technical, and political concerns. This meta-framework can thus serve as a reflective tool for understanding, mapping, and assessing the theory-practice conceptualizations within AI ethics approaches to address and overcome their blind spots.

Determinants for scalable adoption of autonomous AI in the detection of diabetic eye disease in diverse practice types: key best practices learned through collection of real-world data

Autonomous Artificial Intelligence (AI) has the potential to reduce disparities, improve quality of care, and reduce cost by improving access to specialty diagnoses at the point-of-care. Diabetes and related complications represent a significant source of health disparities. Vision loss is a complication of diabetes, and there is extensive evidence supporting annual eye exams for prevention. Prior to the use of autonomous AI, store-and-forward imaging approaches using remote reading centers (asynchronous telemedicine) attempted to increase diabetes related eye exams with limited success. In 2018, after rigorous clinical validation, the first fully autonomous AI system [LumineticsCore™ (formerly IDx-DR), Digital Diagnostics Inc., Coralville, IA, United States] received U.S. Food and Drug Administration (FDA) De Novo authorization. The system diagnoses diabetic retinopathy (including macular edema) without specialist physician overread at the point-of-care. In addition to regulatory clearance, reimbursement, and quality measure updates, successful adoption requires local optimization of the clinical workflow. The general challenges of frontline care clinical workflow have been well documented in the literature. Because healthcare AI is so new, there remains a gap in the literature about challenges and opportunities to embed diagnostic AI into the clinical workflow. The goal of this review is to identify common workflow themes leading to successful adoption, measured as attainment number of exams per month using the autonomous AI system against targets set for each health center. We characterized the workflow in four different US health centers over a 12-month period. Health centers were geographically dispersed across the Midwest, Southwest, Northeast, and West Coast and varied distinctly in terms of size, staffing, resources, financing and demographics of patient populations. After 1 year, the aggregated number of diabetes-related exams per month increased from 89 after the first month of initial deployment to 174 across all sites. Across the diverse practice types, three primary determinants underscored sustainable adoption: (1) Inclusion of Executive and Clinical Champions; (2) Underlining Health Center Resources; and (3) Clinical workflows that contemplate patient identification (pre-visit), LumineticsCore Exam Capture and Provider Consult (patient visit), and Timely Referral Triage (post-visit). In addition to regulatory clearance, reimbursement and quality measures, our review shows that addressing the core determinants for workflow optimization is an essential part of large-scale adoption of innovation. These best practices can be generalizable to other autonomous AI systems in front-line care settings, thereby increasing patient access, improving quality of care, and addressing health disparities.

Artificial intelligence in cardiovascular prevention: new ways will open new doors

Prevention and effective treatment of cardiovascular disease are progressive issues that grow in tandem with the average age of the world population. Over recent decades, the potential role of artificial intelligence in cardiovascular medicine has been increasingly recognized because of the incredible amount of real-world data (RWD) regarding patient health status and healthcare delivery that can be collated from a variety of sources wherein patient information is routinely collected, including patient registries, clinical case reports, reimbursement claims and billing reports, medical devices, and electronic health records. Like any other (health) data, RWD can be analysed in accordance with high-quality research methods, and its analysis can deliver valuable patient-centric insights complementing the information obtained from conventional clinical trials. Artificial intelligence application on RWD has the potential to detect a patient's health trajectory leading to personalized medicine and tailored treatment. This article reviews the benefits of artificial intelligence in cardiovascular prevention and management, focusing on diagnostic and therapeutic improvements without neglecting the limitations of this new scientific approach.

Machine learning and artificial intelligence in research and healthcare

Artificial intelligence (AI) is a broad term referring to the application of computational algorithms that can analyze large data sets to classify, predict, or gain useful conclusions. Under the umbrella of AI is machine learning (ML). ML is the process of building or learning statistical models using previously observed real world data to predict outcomes, or categorize observations based on 'training' provided by humans. These predictions are then applied to future data, all the while folding in the new data into its perpetually improving and calibrated statistical model. The future of AI and ML in healthcare research is exciting and expansive. AI and ML are becoming cornerstones in the medical and healthcare-research domains and are integral in our continued processing and capitalization of robust patient EMR data. Considerations for the use and application of ML in healthcare settings include assessing the quality of data inputs and decision-making that serve as the foundations of the ML model, ensuring the end-product is interpretable, transparent, and ethical concerns are considered throughout the development process. The current and future applications of ML include improving the quality and quantity of data collected from EMRs to improve registry data, utilizing these robust datasets to improve and standardized research protocols and outcomes, clinical decision-making applications, natural language processing and improving the fundamentals of value-based care, to name only a few.

A framework to identify ethical concerns with ML-guided care workflows: a case study of mortality prediction to guide advance care planning

OBJECTIVE

Identifying ethical concerns with ML applications to healthcare (ML-HCA) before problems arise is now a stated goal of ML design oversight groups and regulatory agencies. Lack of accepted standard methodology for ethical analysis, however, presents challenges. In this case study, we evaluate use of a stakeholder "values-collision" approach to identify consequential ethical challenges associated with an ML-HCA for advanced care planning (ACP). Identification of ethical challenges could guide revision and improvement of the ML-HCA.

MATERIALS AND METHODS

We conducted semistructured interviews of the designers, clinician-users, affiliated administrators, and patients, and inductive qualitative analysis of transcribed interviews using modified grounded theory.

RESULTS

Seventeen stakeholders were interviewed. Five "values-collisions"-where stakeholders disagreed about decisions with ethical implications-were identified: (1) end-of-life workflow and how model output is introduced; (2) which stakeholders receive predictions; (3) benefit-harm trade-offs; (4) whether the ML design team has a fiduciary relationship to patients and clinicians; and, (5) how and if to protect early deployment research from external pressures, like news scrutiny, before research is completed.

DISCUSSION

From these findings, the ML design team prioritized: (1) alternative workflow implementation strategies; (2) clarification that prediction was only evaluated for ACP need, not other mortality-related ends; and (3) shielding research from scrutiny until endpoint driven studies were completed.

CONCLUSION

In this case study, our ethical analysis of this ML-HCA for ACP was able to identify multiple sites of intrastakeholder disagreement that mark areas of ethical and value tension. These findings provided a useful initial ethical screening.

Ebenen der Explizierbarkeit für medizinische künstliche Intelligenz: Was brauchen wir normativ und was können wir technisch erreichen?

Definition of the problem

The umbrella term “explicability” refers to the reduction of opacity of artificial intelligence (AI) systems. These efforts are challenging for medical AI applications because higher accuracy often comes at the cost of increased opacity. This entails ethical tensions because physicians and patients desire to trace how results are produced without compromising the performance of AI systems. The centrality of explicability within the informed consent process for medical AI systems compels an ethical reflection on the trade-offs. Which levels of explicability are needed to obtain informed consent when utilizing medical AI?

Arguments

We proceed in five steps: First, we map the terms commonly associated with explicability as described in the ethics and computer science literature, i.e., disclosure, intelligibility, interpretability, and explainability. Second, we conduct a conceptual analysis of the ethical requirements for explicability when it comes to informed consent. Third, we distinguish hurdles for explicability in terms of epistemic and explanatory opacity. Fourth, this then allows to conclude the level of explicability physicians must reach and what patients can expect. In a final step, we show how the identified levels of explicability can technically be met from the perspective of computer science. Throughout our work, we take diagnostic AI systems in radiology as an example.

Conclusion

We determined four levels of explicability that need to be distinguished for ethically defensible informed consent processes and showed how developers of medical AI can technically meet these requirements.

Artificial Intelligence Applications in Clinical Chemistry

Artificial intelligence (AI) applications are an area of active investigation in clinical chemistry. Numerous publications have demonstrated the promise of AI across all phases of testing including preanalytic, analytic, and postanalytic phases; this includes novel methods for detecting common specimen collection errors, predicting laboratory results and diagnoses, and enhancing autoverification workflows. Although AI applications pose several ethical and operational challenges, these technologies are expected to transform the practice of the clinical chemistry laboratory in the near future.

Restoring the missing person to personalized medicine and precision psychiatry

Precision psychiatry has emerged as part of the shift to personalized medicine and builds on frameworks such as the U.S. National Institute of Mental Health Research Domain Criteria (RDoC), multilevel biological "omics" data and, most recently, computational psychiatry. The shift is prompted by the realization that a one-size-fits all approach is inadequate to guide clinical care because people differ in ways that are not captured by broad diagnostic categories. One of the first steps in developing this personalized approach to treatment was the use of genetic markers to guide pharmacotherapeutics based on predictions of pharmacological response or non-response, and the potential risk of adverse drug reactions. Advances in technology have made a greater degree of specificity or precision potentially more attainable. To date, however, the search for precision has largely focused on biological parameters. Psychiatric disorders involve multi-level dynamics that require measures of phenomenological, psychological, behavioral, social structural, and cultural dimensions. This points to the need to develop more fine-grained analyses of experience, self-construal, illness narratives, interpersonal interactional dynamics, and social contexts and determinants of health. In this paper, we review the limitations of precision psychiatry arguing that it cannot reach its goal if it does not include core elements of the processes that give rise to psychopathological states, which include the agency and experience of the person. Drawing from contemporary systems biology, social epidemiology, developmental psychology, and cognitive science, we propose a cultural-ecosocial approach to integrating precision psychiatry with person-centered care.

A cloud-based pipeline for analysis of FHIR and long-read data

Motivation

As genome sequencing becomes cheaper and more accurate, it is becoming increasingly viable to merge this data with electronic health information to inform clinical decisions.

Results

In this work, we demonstrate a full pipeline for working with both PacBio sequencing data and clinical FHIR^® data, from initial data to tertiary analysis. The electronic health records are stored in FHIR^® (Fast Healthcare Interoperability Resource) format, the current leading standard for healthcare data exchange. For the genomic data, we perform variant calling on long-read PacBio HiFi data using Cromwell on Azure. Both data formats are parsed, processed and merged in a single scalable pipeline which securely performs tertiary analyses using cloud-based Jupyter notebooks. We include three example applications: exporting patient information to a database, clustering patients and performing a simple pharmacogenomic study.

Availability and implementation

https://github.com/microsoft/genomicsnotebook/tree/main/fhirgenomics.

Supplementary information

Supplementary data are available at Bioinformatics Advances online.

"Just" accuracy? Procedural fairness demands explainability in AI-based medical resource allocations

The increasing application of artificial intelligence (AI) to healthcare raises both hope and ethical concerns. Some advanced machine learning methods provide accurate clinical predictions at the expense of a significant lack of explainability. Alex John London has defended that accuracy is a more important value than explainability in AI medicine. In this article, we locate the trade-off between accurate performance and explainable algorithms in the context of distributive justice. We acknowledge that accuracy is cardinal from outcome-oriented justice because it helps to maximize patients' benefits and optimizes limited resources. However, we claim that the opaqueness of the algorithmic black box and its absence of explainability threatens core commitments of procedural fairness such as accountability, avoidance of bias, and transparency. To illustrate this, we discuss liver transplantation as a case of critical medical resources in which the lack of explainability in AI-based allocation algorithms is procedurally unfair. Finally, we provide a number of ethical recommendations for when considering the use of unexplainable algorithms in the distribution of health-related resources.

A new diagnostic method for chronic obstructive pulmonary disease using the photoplethysmography signal and hybrid artificial intelligence

Background and Purpose

Chronic obstructive pulmonary disease (COPD), is a primary public health issue globally and in our country, which continues to increase due to poor awareness of the disease and lack of necessary preventive measures. COPD is the result of a blockage of the air sacs known as alveoli within the lungs; it is a persistent sickness that causes difficulty in breathing, cough, and shortness of breath. COPD is characterized by breathing signs and symptoms and airflow challenge because of anomalies in the airways and alveoli that occurs as the result of significant exposure to harmful particles and gases. The spirometry test (breath measurement test), used for diagnosing COPD, is creating difficulties in reaching hospitals, especially in patients with disabilities or advanced disease and in children. To facilitate the diagnostic treatment and prevent these problems, it is far evaluated that using photoplethysmography (PPG) signal in the diagnosis of COPD disease would be beneficial in order to simplify and speed up the diagnosis process and make it more convenient for monitoring. A PPG signal includes numerous components, including volumetric changes in arterial blood that are related to heart activity, fluctuations in venous blood volume that modify the PPG signal, a direct current (DC) component that shows the optical properties of the tissues, and modest energy changes in the body. PPG has typically received the usage of a pulse oximeter, which illuminates the pores and skin and measures adjustments in mild absorption. PPG occurring with every heart rate is an easy signal to measure. PPG signal is modeled by machine learning to predict COPD.

Methods

During the studies, the PPG signal was cleaned of noise, and a brand-new PPG signal having three low-frequency bands of the PPG was obtained. Each of the four signals extracted 25 features. An aggregate of 100 features have been extracted. Additionally, weight, height, and age were also used as characteristics. In the feature selection process, we employed the Fisher method. The intention of using this method is to improve performance.

Results

This improved PPG prediction models have an accuracy rate of 0.95 performance value for all individuals. Classification algorithms used in feature selection algorithm has contributed to a performance increase.

Conclusion

According to the findings, PPG-based COPD prediction models are suitable for usage in practice.

Digital Health Equity: Addressing Power, Usability, and Trust to Strengthen Health Systems

BACKGROUND

Without specific attention to health equity considerations in design, implementation, and evaluation, the rapid expansion of digital health approaches threatens to exacerbate rather than ameliorate existing health disparities.

METHODS

We explored known factors that increase digital health inequity to contextualize the need for equity-centered informatics. This work used a narrative review method to summarize issues about inequities in digital health and to discuss future directions for researchers and clinicians. We searched literature using a combination of relevant keywords (e.g., "digital health", "health equity", etc.) using PubMed and Google Scholar.

RESULTS

We have highlighted strategies for addressing medical marginalization in informatics according to vectors of power such as race and ethnicity, gender identity and modality, sexuality, disability, housing status, citizenship status, and criminalization status.

CONCLUSIONS

We have emphasized collaboration with user and patient groups to define priorities, ensure accessibility and localization, and consider risks in development and utilization of digital health tools. Additionally, we encourage consideration of potential pitfalls in adopting these diversity, equity, and inclusion (DEI)-related strategies.

Using model explanations to guide deep learning models towards consistent explanations for EHR data

It has been shown that identical deep learning (DL) architectures will produce distinct explanations when trained with different hyperparameters that are orthogonal to the task (e.g. random seed, training set order). In domains such as healthcare and finance, where transparency and explainability is paramount, this can be a significant barrier to DL adoption. In this study we present a further analysis of explanation (in)consistency on 6 tabular datasets/tasks, with a focus on Electronic Health Records data. We propose a novel deep learning ensemble architecture that trains its sub-models to produce consistent explanations, improving explanation consistency by as much as 315% (e.g. from 0.02433 to 0.1011 on MIMIC-IV), and on average by 124% (e.g. from 0.12282 to 0.4450 on the BCW dataset). We evaluate the effectiveness of our proposed technique and discuss the implications our results have for both industrial applications of DL and explainability as well as future methodological work.

Ethical Conundrums in the Application of Artificial Intelligence (AI) in Healthcare-A Scoping Review of Reviews

BACKGROUND

With the availability of extensive health data, artificial intelligence has an inordinate capability to expedite medical explorations and revamp healthcare.Artificial intelligence is set to reform the practice of medicine soon. Despite the mammoth advantages of artificial intelligence in the medical field, there exists inconsistency in the ethical and legal framework for the application of AI in healthcare. Although research has been conducted by various medical disciplines investigating the ethical implications of artificial intelligence in the healthcare setting, the literature lacks a holistic approach.

OBJECTIVE

The purpose of this review is to ascertain the ethical concerns of AI applications in healthcare, to identify the knowledge gaps and provide recommendations for an ethical and legal framework.

METHODOLOGY

Electronic databases Pub Med and Google Scholar were extensively searched based on the search strategy pertaining to the purpose of this review. Further screening of the included articles was done on the grounds of the inclusion and exclusion criteria.

RESULTS

The search yielded a total of 1238 articles, out of which 16 articles were identified to be eligible for this review. The selection was strictly based on the inclusion and exclusion criteria mentioned in the manuscript.

CONCLUSION

Artificial intelligence (AI) is an exceedingly puissant technology, with the prospect of advancing medical practice in the years to come. Nevertheless, AI brings with it a colossally abundant number of ethical and legal problems associated with its application in healthcare. There are manifold stakeholders in the legal and ethical issues revolving around AI and medicine. Thus, a multifaceted approach involving policymakers, developers, healthcare providers and patients is crucial to arrive at a feasible solution for mitigating the legal and ethical problems pertaining to AI in healthcare.

Randomised controlled trials in medical AI: ethical considerations

In recent years, there has been a surge of high-profile publications on applications of artificial intelligence (AI) systems for medical diagnosis and prognosis. While AI provides various opportunities for medical practice, there is an emerging consensus that the existing studies show considerable deficits and are unable to establish the clinical benefit of AI systems. Hence, the view that the clinical benefit of AI systems needs to be studied in clinical trials-particularly randomised controlled trials (RCTs)-is gaining ground. However, an issue that has been overlooked so far in the debate is that, compared with drug RCTs, AI RCTs require methodological adjustments, which entail ethical challenges. This paper sets out to develop a systematic account of the ethics of AI RCTs by focusing on the moral principles of clinical equipoise, informed consent and fairness. This way, the objective is to animate further debate on the (research) ethics of medical AI.

Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review

Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.

Explainable, trustworthy, and ethical machine learning for healthcare: A survey

With the advent of machine learning (ML) and deep learning (DL) empowered applications for critical applications like healthcare, the questions about liability, trust, and interpretability of their outputs are raising. The black-box nature of various DL models is a roadblock to clinical utilization. Therefore, to gain the trust of clinicians and patients, we need to provide explanations about the decisions of models. With the promise of enhancing the trust and transparency of black-box models, researchers are in the phase of maturing the field of eXplainable ML (XML). In this paper, we provided a comprehensive review of explainable and interpretable ML techniques for various healthcare applications. Along with highlighting security, safety, and robustness challenges that hinder the trustworthiness of ML, we also discussed the ethical issues arising because of the use of ML/DL for healthcare. We also describe how explainable and trustworthy ML can resolve all these ethical problems. Finally, we elaborate on the limitations of existing approaches and highlight various open research problems that require further development.

Leveraging law and ethics to promote safe and reliable AI/ML in healthcare

Artificial intelligence and machine learning (AI/ML) is poised to disrupt the structure and delivery of healthcare, promising to optimize care clinical care delivery and information management. AI/ML offers potential benefits in healthcare, such as creating novel clinical decision support tools, pattern recognition software, and predictive modeling systems. This raises questions about how AI/ML will impact the physician-patient relationship and the practice of medicine. Effective utilization and reliance on AI/ML also requires that these technologies are safe and reliable. Potential errors could not only pose serious risks to patient safety, but also expose physicians, hospitals, and AI/ML manufacturers to liability. This review describes how the law provides a mechanism to promote safety and reliability of AI/ML systems. On the front end, the Food and Drug Administration (FDA) intends to regulate many AI/ML as medical devices, which corresponds to a set of regulatory requirements prior to product marketing and use. Post-development, a variety of mechanisms in the law provide guardrails for careful deployment into clinical practice that can also incentivize product improvement. This review provides an overview of potential areas of liability arising from AI/ML including malpractice, informed consent, corporate liability, and products liability. Finally, this review summarizes strategies to minimize risk and promote safe and reliable AI/ML.

Trustworthy AI: Closing the gap between development and integration of AI systems in ophthalmic practice

An increasing number of artificial intelligence (AI) systems are being proposed in ophthalmology, motivated by the variety and amount of clinical and imaging data, as well as their potential benefits at the different stages of patient care. Despite achieving close or even superior performance to that of experts, there is a critical gap between development and integration of AI systems in ophthalmic practice. This work focuses on the importance of trustworthy AI to close that gap. We identify the main aspects or challenges that need to be considered along the AI design pipeline so as to generate systems that meet the requirements to be deemed trustworthy, including those concerning accuracy, resiliency, reliability, safety, and accountability. We elaborate on mechanisms and considerations to address those aspects or challenges, and define the roles and responsibilities of the different stakeholders involved in AI for ophthalmic care, i.e., AI developers, reading centers, healthcare providers, healthcare institutions, ophthalmological societies and working groups or committees, patients, regulatory bodies, and payers. Generating trustworthy AI is not a responsibility of a sole stakeholder. There is an impending necessity for a collaborative approach where the different stakeholders are represented along the AI design pipeline, from the definition of the intended use to post-market surveillance after regulatory approval. This work contributes to establish such multi-stakeholder interaction and the main action points to be taken so that the potential benefits of AI reach real-world ophthalmic settings.

A Delphi consensus statement for digital surgery

The use of digital technology is increasing rapidly across surgical specialities, yet there is no consensus for the term 'digital surgery'. This is critical as digital health technologies present technical, governance, and legal challenges which are unique to the surgeon and surgical patient. We aim to define the term digital surgery and the ethical issues surrounding its clinical application, and to identify barriers and research goals for future practice. 38 international experts, across the fields of surgery, AI, industry, law, ethics and policy, participated in a four-round Delphi exercise. Issues were generated by an expert panel and public panel through a scoping questionnaire around key themes identified from the literature and voted upon in two subsequent questionnaire rounds. Consensus was defined if >70% of the panel deemed the statement important and <30% unimportant. A final online meeting was held to discuss consensus statements. The definition of digital surgery as the use of technology for the enhancement of preoperative planning, surgical performance, therapeutic support, or training, to improve outcomes and reduce harm achieved 100% consensus agreement. We highlight key ethical issues concerning data, privacy, confidentiality and public trust, consent, law, litigation and liability, and commercial partnerships within digital surgery and identify barriers and research goals for future practice. Developers and users of digital surgery must not only have an awareness of the ethical issues surrounding digital applications in healthcare, but also the ethical considerations unique to digital surgery. Future research into these issues must involve all digital surgery stakeholders including patients.

Algorithms for Ethical Decision-Making in the Clinic: A Proof of Concept

Machine intelligence already helps medical staff with a number of tasks. Ethical decision-making, however, has not been handed over to computers. In this proof-of-concept study, we show how an algorithm based on Beauchamp and Childress' prima-facie principles could be employed to advise on a range of moral dilemma situations that occur in medical institutions. We explain why we chose fuzzy cognitive maps to set up the advisory system and how we utilized machine learning to train it. We report on the difficult task of operationalizing the principles of beneficence, non-maleficence and patient autonomy, and describe how we selected suitable input parameters that we extracted from a training dataset of clinical cases. The first performance results are promising, but an algorithmic approach to ethics also comes with several weaknesses and limitations. Should one really entrust the sensitive domain of clinical ethics to machine intelligence?

Machine Learning Approaches for Hospital Acquired Pressure Injuries: A Retrospective Study of Electronic Medical Records

Background

Many machine learning heuristics integrate well with Electronic Medical Record (EMR) systems yet often fail to surpass traditional statistical models for biomedical applications.

Objective

We sought to compare predictive performances of 12 machine learning and traditional statistical techniques to predict the occurrence of Hospital Acquired Pressure Injuries (HAPI).

Methods

EMR information was collected from 57,227 hospitalizations acquired from Dartmouth Hitchcock Medical Center (April 2011 to December 2016). Twelve classification algorithms, chosen based upon classic regression and recent machine learning techniques, were trained to predict HAPI incidence and performance was assessed using the Area Under the Receiver Operating Characteristic Curve (AUC).

Results

Logistic regression achieved a performance (AUC = 0.91 ± 0.034) comparable to the other machine learning approaches. We report discordance between machine learning derived predictors compared to the traditional statistical model. We visually assessed important patient-specific factors through Shapley Additive Explanations.

Conclusions

Machine learning models will continue to inform clinical decision-making processes but should be compared to traditional modeling approaches to ensure proper utilization. Disagreements between important predictors found by traditional and machine learning modeling approaches can potentially confuse clinicians and need to be reconciled. These developments represent important steps forward in developing real-time predictive models that can be integrated into EMR systems to reduce unnecessary harm.

The Future Ethics of Artificial Intelligence in Medicine: Making Sense of Collaborative Models

This article examines the role of medical doctors, AI designers, and other stakeholders in making applied AI and machine learning ethically acceptable on the general premises of shared decision-making in medicine. Recent policy documents such as the EU strategy on trustworthy AI and the research literature have often suggested that AI could be made ethically acceptable by increased collaboration between developers and other stakeholders. The article articulates and examines four central alternative models of how AI can be designed and applied in patient care, which we call the ordinary evidence model, the ethical design model, the collaborative model, and the public deliberation model. We argue that the collaborative model is the most promising for covering most AI technology, while the public deliberation model is called for when the technology is recognized as fundamentally transforming the conditions for ethical shared decision-making.

Artificial Intelligence in Emergency Medicine: Benefits, Risks, and Recommendations

BACKGROUND

Artificial intelligence (AI) can be described as the use of computers to perform tasks that formerly required human cognition. The American Medical Association prefers the term 'augmented intelligence' over 'artificial intelligence' to emphasize the assistive role of computers in enhancing physician skills as opposed to replacing them. The integration of AI into emergency medicine, and clinical practice at large, has increased in recent years, and that trend is likely to continue.

DISCUSSION

AI has demonstrated substantial potential benefit for physicians and patients. These benefits are transforming the therapeutic relationship from the traditional physician-patient dyad into a triadic doctor-patient-machine relationship. New AI technologies, however, require careful vetting, legal standards, patient safeguards, and provider education. Emergency physicians (EPs) should recognize the limits and risks of AI as well as its potential benefits.

CONCLUSIONS

EPs must learn to partner with, not capitulate to, AI. AI has proven to be superior to, or on a par with, certain physician skills, such as interpreting radiographs and making diagnoses based on visual cues, such as skin cancer. AI can provide cognitive assistance, but EPs must interpret AI results within the clinical context of individual patients. They must also advocate for patient confidentiality, professional liability coverage, and the essential role of specialty-trained EPs.

Legal and Ethical Consideration in Artificial Intelligence in Healthcare: Who Takes Responsibility?

The legal and ethical issues that confront society due to Artificial Intelligence (AI) include privacy and surveillance, bias or discrimination, and potentially the philosophical challenge is the role of human judgment. Concerns about newer digital technologies becoming a new source of inaccuracy and data breaches have arisen as a result of its use. Mistakes in the procedure or protocol in the field of healthcare can have devastating consequences for the patient who is the victim of the error. Because patients come into contact with physicians at moments in their lives when they are most vulnerable, it is crucial to remember this. Currently, there are no well-defined regulations in place to address the legal and ethical issues that may arise due to the use of artificial intelligence in healthcare settings. This review attempts to address these pertinent issues highlighting the need for algorithmic transparency, privacy, and protection of all the beneficiaries involved and cybersecurity of associated vulnerabilities.

Assessment of Machine Learning-Based Medical Directives to Expedite Care in Pediatric Emergency Medicine

Importance

Increased wait times and long lengths of stay in emergency departments (EDs) are associated with poor patient outcomes. Systems to improve ED efficiency would be useful. Specifically, minimizing the time to diagnosis by developing novel workflows that expedite test ordering can help accelerate clinical decision-making.

Objective

To explore the use of machine learning-based medical directives (MLMDs) to automate diagnostic testing at triage for patients with common pediatric ED diagnoses.

Design, Setting, and Participants

Machine learning models trained on retrospective electronic health record data were evaluated in a decision analytical model study conducted at the ED of the Hospital for Sick Children Toronto, Canada. Data were collected on all patients aged 0 to 18 years presenting to the ED from July 1, 2018, to June 30, 2019 (77 219 total patient visits).

Exposure

Machine learning models were trained to predict the need for urinary dipstick testing, electrocardiogram, abdominal ultrasonography, testicular ultrasonography, bilirubin level testing, and forearm radiographs.

Main Outcomes and Measures

Models were evaluated using area under the receiver operator curve, true-positive rate, false-positive rate, and positive predictive values. Model decision thresholds were determined to limit the total number of false-positive results and achieve high positive predictive values. The time difference between patient triage completion and test ordering was assessed for each use of MLMD. Error rates were analyzed to assess model bias. In addition, model explainability was determined using Shapley Additive Explanations values.

Results

There was a total of 42 238 boys (54.7%) included in model development; mean (SD) age of the children was 5.4 (4.8) years. Models obtained high area under the receiver operator curve (0.89-0.99) and positive predictive values (0.77-0.94) across each of the use cases. The proposed implementation of MLMDs would streamline care for 22.3% of all patient visits and make test results available earlier by 165 minutes (weighted mean) per affected patient. Model explainability for each MLMD demonstrated clinically relevant features having the most influence on model predictions. Models also performed with minimal to no sex bias.

Conclusions and Relevance

The findings of this study suggest the potential for clinical automation using MLMDs. When integrated into clinical workflows, MLMDs may have the potential to autonomously order common ED tests early in a patient's visit with explainability provided to patients and clinicians.

Foundational Considerations for Artificial Intelligence Using Ophthalmic Images

IMPORTANCE

The development of artificial intelligence (AI) and other machine diagnostic systems, also known as software as a medical device, and its recent introduction into clinical practice requires a deeply rooted foundation in bioethics for consideration by regulatory agencies and other stakeholders around the globe.

OBJECTIVES

To initiate a dialogue on the issues to consider when developing a bioethically sound foundation for AI in medicine, based on images of eye structures, for discussion with all stakeholders.

EVIDENCE REVIEW

The scope of the issues and summaries of the discussions under consideration by the Foundational Principles of Ophthalmic Imaging and Algorithmic Interpretation Working Group, as first presented during the Collaborative Community on Ophthalmic Imaging inaugural meeting on September 7, 2020, and afterward in the working group.

FINDINGS

Artificial intelligence has the potential to improve health care access and patient outcome fundamentally while decreasing disparities, lowering cost, and enhancing the care team. Nevertheless, substantial concerns exist. Bioethicists, AI algorithm experts, as well as the Food and Drug Administration and other regulatory agencies, industry, patient advocacy groups, clinicians and their professional societies, other provider groups, and payors (i.e., stakeholders) working together in collaborative communities to resolve the fundamental ethical issues of nonmaleficence, autonomy, and equity are essential to attain this potential. Resolution impacts all levels of the design, validation, and implementation of AI in medicine. Design, validation, and implementation of AI warrant meticulous attention.

CONCLUSIONS AND RELEVANCE

The development of a bioethically sound foundation may be possible if it is based in the fundamental ethical principles of nonmaleficence, autonomy, and equity for considerations for the design, validation, and implementation for AI systems. Achieving such a foundation will be helpful for continuing successful introduction into medicine before consideration by regulatory agencies. Important improvements in accessibility and quality of health care, decrease in health disparities, and lower cost thereby can be achieved. These considerations should be discussed with all stakeholders and expanded on as a useful initiation of this dialogue.

Guidelines for artificial intelligence in medicine: A literature review and content analysis of frameworks (Preprint)

Background

Artificial intelligence (AI) is rapidly expanding in medicine despite a lack of consensus on its application and evaluation.

Objective

We sought to identify current frameworks guiding the application and evaluation of AI for predictive analytics in medicine and to describe the content of these frameworks. We also assessed what stages along the AI translational spectrum (ie, AI development, reporting, evaluation, implementation, and surveillance) the content of each framework has been discussed.

Methods

We performed a literature review of frameworks regarding the oversight of AI in medicine. The search included key topics such as “artificial intelligence,” “machine learning,” “guidance as topic,” and “translational science,” and spanned the time period 2014-2022. Documents were included if they provided generalizable guidance regarding the use or evaluation of AI in medicine. Included frameworks are summarized descriptively and were subjected to content analysis. A novel evaluation matrix was developed and applied to appraise the frameworks’ coverage of content areas across translational stages.

Results

Fourteen frameworks are featured in the review, including six frameworks that provide descriptive guidance and eight that provide reporting checklists for medical applications of AI. Content analysis revealed five considerations related to the oversight of AI in medicine across frameworks: transparency, reproducibility, ethics, effectiveness, and engagement. All frameworks include discussions regarding transparency, reproducibility, ethics, and effectiveness, while only half of the frameworks discuss engagement. The evaluation matrix revealed that frameworks were most likely to report AI considerations for the translational stage of development and were least likely to report considerations for the translational stage of surveillance.

Conclusions

Existing frameworks for the application and evaluation of AI in medicine notably offer less input on the role of engagement in oversight and regarding the translational stage of surveillance. Identifying and optimizing strategies for engagement are essential to ensure that AI can meaningfully benefit patients and other end users.

Mobilizing artificial intelligence to cardiac telerehabilitation

Cardiac telerehabilitation is a method that uses digital technologies to deliver cardiac rehabilitation from a distance. It has been shown to have benefits to improve patients' disease outcomes and quality of life, and further reduce readmission and adverse cardiac events. The outbreak of the coronavirus pandemic has brought considerable new challenges to cardiac rehabilitation, which foster cardiac telerehabilitation to be broadly applied. This transformation is associated with some difficulties that urgently need some innovations to search for the right path. Artificial intelligence, which has a high level of data mining and interpretation, may provide a potential solution. This review evaluates the current application and limitations of artificial intelligence in cardiac telerehabilitation and offers prospects.

Conceptualising fairness: three pillars for medical algorithms and health equity

OBJECTIVES

Fairness is a core concept meant to grapple with different forms of discrimination and bias that emerge with advances in Artificial Intelligence (eg, machine learning, ML). Yet, claims to fairness in ML discourses are often vague and contradictory. The response to these issues within the scientific community has been technocratic. Studies either measure (mathematically) competing definitions of fairness, and/or recommend a range of governance tools (eg, fairness checklists or guiding principles). To advance efforts to operationalise fairness in medicine, we synthesised a broad range of literature.

METHODS

We conducted an environmental scan of English language literature on fairness from 1960-July 31, 2021. Electronic databases Medline, PubMed and Google Scholar were searched, supplemented by additional hand searches. Data from 213 selected publications were analysed using rapid framework analysis. Search and analysis were completed in two rounds: to explore previously identified issues (a priori), as well as those emerging from the analysis (de novo).

RESULTS

Our synthesis identified 'Three Pillars for Fairness': transparency, impartiality and inclusion. We draw on these insights to propose a multidimensional conceptual framework to guide empirical research on the operationalisation of fairness in healthcare.

DISCUSSION

We apply the conceptual framework generated by our synthesis to risk assessment in psychiatry as a case study. We argue that any claim to fairness must reflect critical assessment and ongoing social and political deliberation around these three pillars with a range of stakeholders, including patients.

CONCLUSION

We conclude by outlining areas for further research that would bolster ongoing commitments to fairness and health equity in healthcare.

Nanobioengineering: A promising approach for early detection of COVID-19

Unique pneumonia due to an unknown source emerged in December 2019 in the city of Wuhan, China. Consequently, the World Health Organization (WHO) declared this condition as a new coronavirus disease-19 also known as COVID-19 on February 11, 2020, which on March 13, 2020 was declared as a pandemic. The virus that causes COVID-19 was found to have a similar genome (80% similarity) with the previously known acute respiratory syndrome also known as SARS-CoV. The novel virus was later named Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). SARS-CoV-2 falls in the family of Coronaviridae which is further divided into Nidovirales and another subfamily called Orthocoronavirinae. The four generations of the coronaviruses belongs to the Orthocoronavirinae family that consists of alpha, beta, gamma and delta coronavirus which are denoted as α-CoV, β-CoV, γ-CoV, δ-CoV respectively. The α-CoV and β-CoVs are mainly known to infect mammals whereas γ-CoV and δ-CoV are generally found in birds. The β-CoVs also comprise of SARS-CoV and also include another virus that was found in the Middle East called the Middle East respiratory syndrome virus (MERS-CoV) and the cause of current pandemic SARS-CoV-2. These viruses initially cause the development of pneumonia in the patients and further development of a severe case of acute respiratory distress syndrome (ARDS) and other related symptoms that can be fatal leading to death.

Pre-injury performance is most important for predicting the level of match participation after Achilles tendon ruptures in elite soccer players: a study using a machine learning classifier

PURPOSE

Achilles tendon ruptures (ATR) are career-threatening injuries in elite soccer players due to the decreased sports performance they commonly inflict. This study presents an exploratory data analysis of match participation before and after ATRs and an evaluation of the performance of a machine learning (ML) model based on pre-injury features to predict whether a player will return to a previous level of match participation.

METHODS

The website transfermarkt.com was mined, between January and March of 2021, for relevant entries regarding soccer players who suffered an ATR while playing in first or second leagues. The difference between average minutes played per match (MPM) 1 year before injury and between 1 and 2 years after the injury was used to identify patterns in match participation after injury. Clustering analysis was performed using k-means clustering. Predictions of post-injury match participation were made using the XGBoost classification algorithm. The performance of this model was evaluated using the area under the receiver operating characteristic curve (AUROC) and Brier score loss (BSL).

RESULTS

Two hundred and nine players were included in the study. Data from 32,853 matches was analysed. Exploratory data analysis revealed that forwards, midfielders and defenders increased match participation during the first year after injury, with goalkeepers still improving at 2 years. Players were grouped into four clusters regarding the difference between MPMs 1 year before injury and between 1 and 2 years after the injury. These groups ranged between a severe decrease (n = 34; - 59 ± 13 MPM), moderate decrease (n = 75; - 25 ± 8 MPM), maintenance (n = 70; 0 ± 8 MPM), or increase (n = 30; 32 ± 13 MPM). Regarding the predictive model, the average AUROC after cross-validation was 0.81 ± 0.10, and the BSL was 0.12, with the most important features relating to pre-injury match participation.

CONCLUSION

Most players take 1 year to reach peak match participation after an ATR. Good performance was attained using a ML classifier to predict the level of match participation following an ATR, with features related to pre-injury match participation displaying the highest importance.

LEVEL OF EVIDENCE

I.

Potential applications and performance of machine learning techniques and algorithms in clinical practice: A systematic review

PURPOSE

The advent of clinically adapted machine learning algorithms can solve numerous problems ranging from disease diagnosis and prognosis to therapy recommendations. This systematic review examines the performance of machine learning (ML) algorithms and evaluates the progress made to date towards their implementation in clinical practice.

METHODS

Systematic searching of databases (PubMed, MEDLINE, Scopus, Google Scholar, Cochrane Library and WHO Covid-19 database) to identify original articles published between January 2011 and October 2021. Studies reporting ML techniques in clinical practice involving humans and ML algorithms with a performance metric were considered.

RESULTS

Of 873 unique articles identified, 36 studies were eligible for inclusion. The XGBoost (extreme gradient boosting) algorithm showed the highest potential for clinical applications (n = 7 studies); this was followed jointly by random forest algorithm, logistic regression, and the support vector machine, respectively (n = 5 studies). Prediction of outcomes (n = 33), in particular Inflammatory diseases (n = 7) received the most attention followed by cancer and neuropsychiatric disorders (n = 5 for each) and Covid-19 (n = 4). Thirty-three out of the thirty-six included studies passed more than 50% of the selected quality assessment criteria in the TRIPOD checklist. In contrast, none of the studies could achieve an ideal overall bias rating of 'low' based on the PROBAST checklist. In contrast, only three studies showed evidence of the deployment of ML algorithm(s) in clinical practice.

CONCLUSIONS

ML is potentially a reliable tool for clinical decision support. Although advocated widely in clinical practice, work is still in progress to validate clinically adapted ML algorithms. Improving quality standards, transparency, and interpretability of ML models will further lower the barriers to acceptability.

Trustworthy Artificial Intelligence in Medical Imaging

Trust in artificial intelligence (AI) by society and the development of trustworthy AI systems and ecosystems are critical for the progress and implementation of AI technology in medicine. With the growing use of AI in a variety of medical and imaging applications, it is more vital than ever to make these systems dependable and trustworthy. Fourteen core principles are considered in this article aiming to move the needle more closely to systems that are accurate, resilient, fair, explainable, safe, and transparent: toward trustworthy AI.

Believing in Black Boxes: Machine Learning for Healthcare Does Not Need Explainability to be Evidence-Based

OBJECTIVE

To examine the role of explainability in machine learning for healthcare (MLHC), and its necessity and significance with respect to effective and ethical MLHC application.

STUDY DESIGN AND SETTING

This commentary engages with the growing and dynamic corpus of literature on the use of MLHC and artificial intelligence (AI) in medicine, which provide the context for a focused narrative review of arguments presented in favour of and opposition to explainability in MLHC.

RESULTS

We find that concerns regarding explainability are not limited to MLHC, but rather extend to numerous well-validated treatment interventions as well as to human clinical judgment itself. We examine the role of evidence-based medicine in evaluating inexplicable treatments and technologies, and highlight the analogy between the concept of explainability in MLHC and the related concept of mechanistic reasoning in evidence-based medicine.

CONCLUSION

Ultimately, we conclude that the value of explainability in MLHC is not intrinsic, but is instead instrumental to achieving greater imperatives such as performance and trust. We caution against the uncompromising pursuit of explainability, and advocate instead for the development of robust empirical methods to successfully evaluate increasingly inexplicable algorithmic systems.

What Makes Artificial Intelligence Exceptional in Health Technology Assessment?

The application of artificial intelligence (AI) may revolutionize the healthcare system, leading to enhance efficiency by automatizing routine tasks and decreasing health-related costs, broadening access to healthcare delivery, targeting more precisely patient needs, and assisting clinicians in their decision-making. For these benefits to materialize, governments and health authorities must regulate AI, and conduct appropriate health technology assessment (HTA). Many authors have highlighted that AI health technologies (AIHT) challenge traditional evaluation and regulatory processes. To inform and support HTA organizations and regulators in adapting their processes to AIHTs, we conducted a systematic review of the literature on the challenges posed by AIHTs in HTA and health regulation. Our research question was: What makes artificial intelligence exceptional in HTA? The current body of literature appears to portray AIHTs as being exceptional to HTA. This exceptionalism is expressed along 5 dimensions: 1) AIHT's distinctive features; 2) their systemic impacts on health care and the health sector; 3) the increased expectations towards AI in health; 4) the new ethical, social and legal challenges that arise from deploying AI in the health sector; and 5) the new evaluative constraints that AI poses to HTA. Thus, AIHTs are perceived as exceptional because of their technological characteristics and potential impacts on society at large. As AI implementation by governments and health organizations carries risks of generating new, and amplifying existing, challenges, there are strong arguments for taking into consideration the exceptional aspects of AIHTs, especially as their impacts on the healthcare system will be far greater than that of drugs and medical devices. As AIHTs begin to be increasingly introduced into the health care sector, there is a window of opportunity for HTA agencies and scholars to consider AIHTs' exceptionalism and to work towards only deploying clinically, economically, socially acceptable AIHTs in the health care system.