Organizational readiness for artificial intelligence in health care: insights for decision-making and practice

Implications of Large Language Models for Quality and Efficiency of Neurologic Care: Emerging Issues in Neurology

Large language models (LLMs) are advanced artificial intelligence (AI) systems that excel in recognizing and generating human-like language, possibly serving as valuable tools for neurology-related information tasks. Although LLMs have shown remarkable potential in various areas, their performance in the dynamic environment of daily clinical practice remains uncertain. This article outlines multiple limitations and challenges of using LLMs in clinical settings that need to be addressed, including limited clinical reasoning, variable reliability and accuracy, reproducibility bias, self-serving bias, sponsorship bias, and potential for exacerbating health care disparities. These challenges are further compounded by practical business considerations and infrastructure requirements, including associated costs. To overcome these hurdles and harness the potential of LLMs effectively, this article includes considerations for health care organizations, researchers, and neurologists contemplating the use of LLMs in clinical practice. It is essential for health care organizations to cultivate a culture that welcomes AI solutions and aligns them seamlessly with health care operations. Clear objectives and business plans should guide the selection of AI solutions, ensuring they meet organizational needs and budget considerations. Engaging both clinical and nonclinical stakeholders can help secure necessary resources, foster trust, and ensure the long-term sustainability of AI implementations. Testing, validation, training, and ongoing monitoring are pivotal for successful integration. For neurologists, safeguarding patient data privacy is paramount. Seeking guidance from institutional information technology resources for informed, compliant decisions, and remaining vigilant against biases in LLM outputs are essential practices in responsible and unbiased utilization of AI tools. In research, obtaining institutional review board approval is crucial when dealing with patient data, even if deidentified, to ensure ethical use. Compliance with established guidelines like SPIRIT-AI, MI-CLAIM, and CONSORT-AI is necessary to maintain consistency and mitigate biases in AI research. In summary, the integration of LLMs into clinical neurology offers immense promise while presenting formidable challenges. Awareness of these considerations is vital for harnessing the potential of AI in neurologic care effectively and enhancing patient care quality and safety. The article serves as a guide for health care organizations, researchers, and neurologists navigating this transformative landscape.

Understanding the integration of artificial intelligence in healthcare organisations and systems through the NASSS framework: a qualitative study in a leading Canadian academic centre

BACKGROUND

Artificial intelligence (AI) technologies are expected to "revolutionise" healthcare. However, despite their promises, their integration within healthcare organisations and systems remains limited. The objective of this study is to explore and understand the systemic challenges and implications of their integration in a leading Canadian academic hospital.

METHODS

Semi-structured interviews were conducted with 29 stakeholders concerned by the integration of a large set of AI technologies within the organisation (e.g., managers, clinicians, researchers, patients, technology providers). Data were collected and analysed using the Non-Adoption, Abandonment, Scale-up, Spread, Sustainability (NASSS) framework.

RESULTS

Among enabling factors and conditions, our findings highlight: a supportive organisational culture and leadership leading to a coherent organisational innovation narrative; mutual trust and transparent communication between senior management and frontline teams; the presence of champions, translators, and boundary spanners for AI able to build bridges and trust; and the capacity to attract technical and clinical talents and expertise. Constraints and barriers include: contrasting definitions of the value of AI technologies and ways to measure such value; lack of real-life and context-based evidence; varying patients' digital and health literacy capacities; misalignments between organisational dynamics, clinical and administrative processes, infrastructures, and AI technologies; lack of funding mechanisms covering the implementation, adaptation, and expertise required; challenges arising from practice change, new expertise development, and professional identities; lack of official professional, reimbursement, and insurance guidelines; lack of pre- and post-market approval legal and governance frameworks; diversity of the business and financing models for AI technologies; and misalignments between investors' priorities and the needs and expectations of healthcare organisations and systems.

CONCLUSION

Thanks to the multidimensional NASSS framework, this study provides original insights and a detailed learning base for analysing AI technologies in healthcare from a thorough socio-technical perspective. Our findings highlight the importance of considering the complexity characterising healthcare organisations and systems in current efforts to introduce AI technologies within clinical routines. This study adds to the existing literature and can inform decision-making towards a judicious, responsible, and sustainable integration of these technologies in healthcare organisations and systems.

Suitability of the Current Health Technology Assessment of Innovative Artificial Intelligence-Based Medical Devices: Scoping Literature Review

BACKGROUND

Artificial intelligence (AI)-based medical devices have garnered attention due to their ability to revolutionize medicine. Their health technology assessment framework is lacking.

OBJECTIVE

This study aims to analyze the suitability of each health technology assessment (HTA) domain for the assessment of AI-based medical devices.

METHODS

We conducted a scoping literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We searched databases (PubMed, Embase, and Cochrane Library), gray literature, and HTA agency websites.

RESULTS

A total of 10.1% (78/775) of the references were included. Data quality and integration are vital aspects to consider when describing and assessing the technical characteristics of AI-based medical devices during an HTA process. When it comes to implementing specialized HTA for AI-based medical devices, several practical challenges and potential barriers could be highlighted and should be taken into account (AI technological evolution timeline, data requirements, complexity and transparency, clinical validation and safety requirements, regulatory and ethical considerations, and economic evaluation).

CONCLUSIONS

The adaptation of the HTA process through a methodological framework for AI-based medical devices enhances the comparability of results across different evaluations and jurisdictions. By defining the necessary expertise, the framework supports the development of a skilled workforce capable of conducting robust and reliable HTAs of AI-based medical devices. A comprehensive adapted HTA framework for AI-based medical devices can provide valuable insights into the effectiveness, cost-effectiveness, and societal impact of AI-based medical devices, guiding their responsible implementation and maximizing their benefits for patients and health care systems.

Insights from semi-structured interviews on integrating artificial intelligence in clinical chemistry laboratory practices

BACKGROUND

Artificial intelligence (AI) is gradually transforming the practises of healthcare providers. Over the last two decades, the advent of AI into numerous aspects of pathology has opened transformative possibilities in how we practise laboratory medicine. Objectives of this study were to explore how AI could impact the clinical practices of professionals working in Clinical Chemistry laboratories, while also identifying effective strategies in medical education to facilitate the required changes.

METHODS

From March to August 2022, an exploratory qualitative study was conducted at the Section of Clinical Chemistry, Department of Pathology and Laboratory Medicine, Aga Khan University, Karachi, Pakistan, in collaboration with Keele University, Newcastle, United Kingdom. Semi-structured interviews were conducted to collect information from diverse group of professionals working in Clinical Chemistry laboratories. All interviews were audio recorded and transcribed verbatim. They were asked what changes AI would involve in the laboratory, what resources would be necessary, and how medical education would assist them in adapting to the change. A content analysis was conducted, resulting in the development of codes and themes based on the analyzed data.

RESULTS

The interviews were analysed to identify three primary themes: perspectives and considerations for AI adoption, educational and curriculum adjustments, and implementation techniques. Although the use of diagnostic algorithms is currently limited in Pakistani Clinical Chemistry laboratories, the application of AI is expanding. All thirteen participants stated their reasons for being hesitant to use AI. Participants stressed the importance of critical aspects for effective AI deployment, the need of a collaborative integrative approach, and the need for constant horizon scanning to keep up with AI developments.

CONCLUSIONS

Three primary themes related to AI adoption were identified: perspectives and considerations, educational and curriculum adjustments, and implementation techniques. The study's findings give a sound foundation for making suggestions to clinical laboratories, scientific bodies, and national and international Clinical Chemistry and laboratory medicine organisations on how to manage pathologists' shifting practises because of AI.

Using artificial intelligence to promote equitable care for inpatients with language barriers and complex medical needs: clinical stakeholder perspectives

OBJECTIVES

Inpatients with language barriers and complex medical needs suffer disparities in quality of care, safety, and health outcomes. Although in-person interpreters are particularly beneficial for these patients, they are underused. We plan to use machine learning predictive analytics to reliably identify patients with language barriers and complex medical needs to prioritize them for in-person interpreters.

MATERIALS AND METHODS

This qualitative study used stakeholder engagement through semi-structured interviews to understand the perceived risks and benefits of artificial intelligence (AI) in this domain. Stakeholders included clinicians, interpreters, and personnel involved in caring for these patients or for organizing interpreters. Data were coded and analyzed using NVIVO software.

RESULTS

We completed 49 interviews. Key perceived risks included concerns about transparency, accuracy, redundancy, privacy, perceived stigmatization among patients, alert fatigue, and supply-demand issues. Key perceived benefits included increased awareness of in-person interpreters, improved standard of care and prioritization for interpreter utilization; a streamlined process for accessing interpreters, empowered clinicians, and potential to overcome clinician bias.

DISCUSSION

This is the first study that elicits stakeholder perspectives on the use of AI with the goal of improved clinical care for patients with language barriers. Perceived benefits and risks related to the use of AI in this domain, overlapped with known hazards and values of AI but some benefits were unique for addressing challenges with providing interpreter services to patients with language barriers.

CONCLUSION

Artificial intelligence to identify and prioritize patients for interpreter services has the potential to improve standard of care and address healthcare disparities among patients with language barriers.

Histopathology image classification: highlighting the gap between manual analysis and AI automation

The field of histopathological image analysis has evolved significantly with the advent of digital pathology, leading to the development of automated models capable of classifying tissues and structures within diverse pathological images. Artificial intelligence algorithms, such as convolutional neural networks, have shown remarkable capabilities in pathology image analysis tasks, including tumor identification, metastasis detection, and patient prognosis assessment. However, traditional manual analysis methods have generally shown low accuracy in diagnosing colorectal cancer using histopathological images. This study investigates the use of AI in image classification and image analytics using histopathological images using the histogram of oriented gradients method. The study develops an AI-based architecture for image classification using histopathological images, aiming to achieve high performance with less complexity through specific parameters and layers. In this study, we investigate the complicated state of histopathological image classification, explicitly focusing on categorizing nine distinct tissue types. Our research used open-source multi-centered image datasets that included records of 100.000 non-overlapping images from 86 patients for training and 7180 non-overlapping images from 50 patients for testing. The study compares two distinct approaches, training artificial intelligence-based algorithms and manual machine learning models, to automate tissue classification. This research comprises two primary classification tasks: binary classification, distinguishing between normal and tumor tissues, and multi-classification, encompassing nine tissue types, including adipose, background, debris, stroma, lymphocytes, mucus, smooth muscle, normal colon mucosa, and tumor. Our findings show that artificial intelligence-based systems can achieve 0.91 and 0.97 accuracy in binary and multi-class classifications. In comparison, the histogram of directed gradient features and the Random Forest classifier achieved accuracy rates of 0.75 and 0.44 in binary and multi-class classifications, respectively. Our artificial intelligence-based methods are generalizable, allowing them to be integrated into histopathology diagnostics procedures and improve diagnostic accuracy and efficiency. The CNN model outperforms existing machine learning techniques, demonstrating its potential to improve the precision and effectiveness of histopathology image analysis. This research emphasizes the importance of maintaining data consistency and applying normalization methods during the data preparation stage for analysis. It particularly highlights the potential of artificial intelligence to assess histopathological images.

Critical success factors for creating sustainable digital health applications: A systematic review of the German case

Objective

The Covid-19 pandemic has accelerated the adoption of digital technologies to address social needs, leading to increased investments in digital healthcare applications. Germany implemented a special law called the "Digitales Versorgungsgesetz" (DVG-Digital Supply Act) in 2019, which enables the reimbursement of digital health applications, including digital therapeutics (DTx), through a fast-track process. The Federal Institute for Drugs and Medical Devices (BfArM), the German federal authority responsible for overseeing digital health applications, has implemented legislative adjustments since the law's introduction, which have increased requirements for these applications and potentially led to the removal of some from the directory as well as a slowdown in the addition of new ones. To counteract this trend, this work aimed to identify key success factors for digital health applications (DiGAs).

Methods

This research identifies critical success factors through a structured literature review for developing sustainable digital health applications within the European healthcare systems, specifically DiGAs. The study aims to support the ongoing digital transformation in healthcare.

Results

The identified success factors that significantly impact the sustainability of DiGAs include patient-centered design, application effectiveness, user-friendliness, and adherence to data protection and information security regulations using standardized approaches. These factors are crucial in preventing the failure of DiGA manufacturers in European countries.

Conclusion

By considering and implementing these critical success factors, DiGA manufacturers can enhance their chances of long-term success and contribute to the digital transformation of the healthcare system in Europe.

How Nurses' Person-Organization Fit Influences Organizational Loyalty

Background

High turnover rates among nurses are a global concern due to the shortage of skilled professionals and increasing demand for high-quality healthcare. This study aims to enhance understanding of organizational fit by examining the impact of Person-organization fit (P-O fit) on organizational loyalty through the mediating role of organizational support and service quality, and the moderating impact of role ambiguity.

Methods

Using a convenience sampling technique, we employed a survey methodology by developing a questionnaire. Data were collected from a sample of 614 nurses in five different healthcare sectors in China. Employing SmartPLS 3.3, we conducted a Structural Equation Modeling (SEM) analysis to examine the relationships among the specified variables.

Results

The findings of the structural analysis suggest that the P-O fit influences organizational loyalty in the healthcare sector. Organizational support and service quality were identified as partial mediators of the P-O fit-organizational loyalty link. Additionally, the role of ambiguity represented a negative moderating impact between service quality and organizational loyalty.

Discussion

Overall, the study's findings extend the understanding of person-organization fit, organizational support, service quality, role ambiguity, and organizational loyalty in the context of healthcare sectors and offer implications for medical authorities. Discussions, limitations, practical implications, and suggestions for further research are also provided.

A Review of the Role of Artificial Intelligence in Healthcare

Artificial intelligence (AI) applications have transformed healthcare. This study is based on a general literature review uncovering the role of AI in healthcare and focuses on the following key aspects: (i) medical imaging and diagnostics, (ii) virtual patient care, (iii) medical research and drug discovery, (iv) patient engagement and compliance, (v) rehabilitation, and (vi) other administrative applications. The impact of AI is observed in detecting clinical conditions in medical imaging and diagnostic services, controlling the outbreak of coronavirus disease 2019 (COVID-19) with early diagnosis, providing virtual patient care using AI-powered tools, managing electronic health records, augmenting patient engagement and compliance with the treatment plan, reducing the administrative workload of healthcare professionals (HCPs), discovering new drugs and vaccines, spotting medical prescription errors, extensive data storage and analysis, and technology-assisted rehabilitation. Nevertheless, this science pitch meets several technical, ethical, and social challenges, including privacy, safety, the right to decide and try, costs, information and consent, access, and efficacy, while integrating AI into healthcare. The governance of AI applications is crucial for patient safety and accountability and for raising HCPs' belief in enhancing acceptance and boosting significant health consequences. Effective governance is a prerequisite to precisely address regulatory, ethical, and trust issues while advancing the acceptance and implementation of AI. Since COVID-19 hit the global health system, the concept of AI has created a revolution in healthcare, and such an uprising could be another step forward to meet future healthcare needs.

Are current clinical studies on artificial intelligence-based medical devices comprehensive enough to support a full health technology assessment? A systematic review

INTRODUCTION

Artificial Intelligence-based Medical Devices (AI-based MDs) are experiencing exponential growth in healthcare. This study aimed to investigate whether current studies assessing AI contain the information required for health technology assessment (HTA) by HTA bodies.

METHODS

We conducted a systematic literature review based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses methodology to extract articles published between 2016 and 2021 related to the assessment of AI-based MDs. Data extraction focused on study characteristics, technology, algorithms, comparators, and results. AI quality assessment and HTA scores were calculated to evaluate whether the items present in the included studies were concordant with the HTA requirements. We performed a linear regression for the HTA and AI scores with the explanatory variables of the impact factor, publication date, and medical specialty. We conducted a univariate analysis of the HTA score and a multivariate analysis of the AI score with an alpha risk of 5 %.

RESULTS

Of 5578 retrieved records, 56 were included. The mean AI quality assessment score was 67 %; 32 % of articles had an AI quality score ≥ 70 %, 50 % had a score between 50 % and 70 %, and 18 % had a score under 50 %. The highest quality scores were observed for the study design (82 %) and optimisation (69 %) categories, whereas the scores were lowest in the clinical practice category (23 %). The mean HTA score was 52 % for all seven domains. 100 % of the studies assessed clinical effectiveness, whereas only 9 % evaluated safety, and 20 % evaluated economic issues. There was a statistically significant relationship between the impact factor and the HTA and AI scores (both p = 0.046).

DISCUSSION

Clinical studies on AI-based MDs have limitations and often lack adapted, robust, and complete evidence. High-quality datasets are also required because the output data can only be trusted if the inputs are reliable. The existing assessment frameworks are not specifically designed to assess AI-based MDs. From the perspective of regulatory authorities, we suggest that these frameworks should be adapted to assess the interpretability, explainability, cybersecurity, and safety of ongoing updates. From the perspective of HTA agencies, we highlight that transparency, professional and patient acceptance, ethical issues, and organizational changes are required for the implementation of these devices. Economic assessments of AI should rely on a robust methodology (business impact or health economic models) to provide decision-makers with more reliable evidence.

CONCLUSION

Currently, AI studies are insufficient to cover HTA prerequisites. HTA processes also need to be adapted because they do not consider the important specificities of AI-based MDs. Specific HTA workflows and accurate assessment tools should be designed to standardise evaluations, generate reliable evidence, and create confidence.

Drawbacks of Artificial Intelligence and Their Potential Solutions in the Healthcare Sector

Artificial intelligence (AI) has the potential to make substantial progress toward the goal of making healthcare more personalized, predictive, preventative, and interactive. We believe AI will continue its present path and ultimately become a mature and effective tool for the healthcare sector. Besides this AI-based systems raise concerns regarding data security and privacy. Because health records are important and vulnerable, hackers often target them during data breaches. The absence of standard guidelines for the moral use of AI and ML in healthcare has only served to worsen the situation. There is debate about how far artificial intelligence (AI) may be utilized ethically in healthcare settings since there are no universal guidelines for its use. Therefore, maintaining the confidentiality of medical records is crucial. This study enlightens the possible drawbacks of AI in the implementation of healthcare sector and their solutions to overcome these situations.

Graphical Abstract

Artificial intelligence-enhanced care pathway planning and scheduling system: content validity assessment of required functionalities

BACKGROUND

Artificial intelligence (AI) and machine learning are transforming the optimization of clinical and patient workflows in healthcare. There is a need for research to specify clinical requirements for AI-enhanced care pathway planning and scheduling systems to improve human-AI interaction in machine learning applications. The aim of this study was to assess content validity and prioritize the most relevant functionalities of an AI-enhanced care pathway planning and scheduling system.

METHODS

A prospective content validity assessment was conducted in five university hospitals in three different countries using an electronic survey. The content of the survey was formed from clinical requirements, which were formulated into generic statements of required AI functionalities. The relevancy of each statement was evaluated using a content validity index. In addition, weighted ranking points were calculated to prioritize the most relevant functionalities of an AI-enhanced care pathway planning and scheduling system.

RESULTS

A total of 50 responses were received from clinical professionals from three European countries. An item-level content validity index ranged from 0.42 to 0.96. 45% of the generic statements were considered good. The highest ranked functionalities for an AI-enhanced care pathway planning and scheduling system were related to risk assessment, patient profiling, and resources. The highest ranked functionalities for the user interface were related to the explainability of machine learning models.

CONCLUSION

This study provided a comprehensive list of functionalities that can be used to design future AI-enhanced solutions and evaluate the designed solutions against requirements. The relevance of statements concerning the AI functionalities were considered somewhat relevant, which might be due to the low level or organizational readiness for AI in healthcare.

Design and Implementation of a Comprehensive AI Dashboard for Real-Time Prediction of Adverse Prognosis of ED Patients

The emergency department (ED) is at the forefront of medical care, and the medical team needs to make outright judgments and treatment decisions under time constraints. Thus, knowing how to make personalized and precise predictions is a very challenging task. With the advancement of artificial intelligence (AI) technology, Chi Mei Medical Center (CMMC) adopted AI, the Internet of Things (IoT), and interaction technologies to establish diverse prognosis prediction models for eight diseases based on the ED electronic medical records of three branch hospitals. CMMC integrated these predictive models to form a digital AI dashboard, showing the risk status of all ED patients diagnosed with any of these eight diseases. This study first explored the methodology of CMMC’s AI development and proposed a four-tier AI dashboard architecture for ED implementation. The AI dashboard’s ease of use, usefulness, and acceptance was also strongly affirmed by the ED medical staff. The ED AI dashboard is an effective tool in the implementation of real-time risk monitoring of patients in the ED and could improve the quality of care as a part of best practice. Based on the results of this study, it is suggested that healthcare institutions thoughtfully consider tailoring their ED dashboard designs to adapt to their unique workflows and environments.

Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden

BACKGROUND

Artificial intelligence (AI) for healthcare presents potential solutions to some of the challenges faced by health systems around the world. However, it is well established in implementation and innovation research that novel technologies are often resisted by healthcare leaders, which contributes to their slow and variable uptake. Although research on various stakeholders' perspectives on AI implementation has been undertaken, very few studies have investigated leaders' perspectives on the issue of AI implementation in healthcare. It is essential to understand the perspectives of healthcare leaders, because they have a key role in the implementation process of new technologies in healthcare. The aim of this study was to explore challenges perceived by leaders in a regional Swedish healthcare setting concerning the implementation of AI in healthcare.

METHODS

The study takes an explorative qualitative approach. Individual, semi-structured interviews were conducted from October 2020 to May 2021 with 26 healthcare leaders. The analysis was performed using qualitative content analysis, with an inductive approach.

RESULTS

The analysis yielded three categories, representing three types of challenge perceived to be linked with the implementation of AI in healthcare: 1) Conditions external to the healthcare system; 2) Capacity for strategic change management; 3) Transformation of healthcare professions and healthcare practice.

CONCLUSIONS

In conclusion, healthcare leaders highlighted several implementation challenges in relation to AI within and beyond the healthcare system in general and their organisations in particular. The challenges comprised conditions external to the healthcare system, internal capacity for strategic change management, along with transformation of healthcare professions and healthcare practice. The results point to the need to develop implementation strategies across healthcare organisations to address challenges to AI-specific capacity building. Laws and policies are needed to regulate the design and execution of effective AI implementation strategies. There is a need to invest time and resources in implementation processes, with collaboration across healthcare, county councils, and industry partnerships.

Change Management and Digital Innovations in Hospitals of Five European Countries

The objective of the paper is to evaluate the quality of systemic change management (CHM) and readiness for change in five Central European countries. The secondary goal is to identify trends and upcoming changes in the field of digital innovations in healthcare. The results show that all compared countries (regardless of their historical context) deal with similar CHM challenges with a rather similar degree of success. A questionnaire distributed to hospitals clearly showed that there is still considerable room for improvement in terms of the use of specific CHM tools. A review focused on digital innovations based on the PRISMA statement showed that there are five main directions, namely, data collection and integration, telemedicine, artificial intelligence, electronic medical records, and M-Health. In the hospital environment, there are considerable reservations in applying change management principles, as well as the absence of a systemic approach. The main factors that must be monitored for a successful and sustainable CHM include a clearly defined and widely communicated vision, early engagement of all stakeholders, precisely set rules, adaptation to the local context and culture, provision of a technical base, and a step-by-step implementation with strong feedback.

Adoption of Machine Learning Systems for Medical Diagnostics in Clinics: Qualitative Interview Study

Background

Recently, machine learning (ML) has been transforming our daily lives by enabling intelligent voice assistants, personalized support for purchase decisions, and efficient credit card fraud detection. In addition to its everyday applications, ML holds the potential to improve medicine as well, especially with regard to diagnostics in clinics. In a world characterized by population growth, demographic change, and the global COVID-19 pandemic, ML systems offer the opportunity to make diagnostics more effective and efficient, leading to a high interest of clinics in such systems. However, despite the high potential of ML, only a few ML systems have been deployed in clinics yet, as their adoption process differs significantly from the integration of prior health information technologies given the specific characteristics of ML.

Objective

This study aims to explore the factors that influence the adoption process of ML systems for medical diagnostics in clinics to foster the adoption of these systems in clinics. Furthermore, this study provides insight into how these factors can be used to determine the ML maturity score of clinics, which can be applied by practitioners to measure the clinic status quo in the adoption process of ML systems.

Methods

To gain more insight into the adoption process of ML systems for medical diagnostics in clinics, we conducted a qualitative study by interviewing 22 selected medical experts from clinics and their suppliers with profound knowledge in the field of ML. We used a semistructured interview guideline, asked open-ended questions, and transcribed the interviews verbatim. To analyze the transcripts, we first used a content analysis approach based on the health care–specific framework of nonadoption, abandonment, scale-up, spread, and sustainability. Then, we drew on the results of the content analysis to create a maturity model for ML adoption in clinics according to an established development process.

Results

With the help of the interviews, we were able to identify 13 ML-specific factors that influence the adoption process of ML systems in clinics. We categorized these factors according to 7 domains that form a holistic ML adoption framework for clinics. In addition, we created an applicable maturity model that could help practitioners assess their current state in the ML adoption process.

Conclusions

Many clinics still face major problems in adopting ML systems for medical diagnostics; thus, they do not benefit from the potential of these systems. Therefore, both the ML adoption framework and the maturity model for ML systems in clinics can not only guide future research that seeks to explore the promises and challenges associated with ML systems in a medical setting but also be a practical reference point for clinicians.

A survey of extant organizational and computational setups for deploying predictive models in health systems

OBJECTIVE

Artificial intelligence (AI) and machine learning (ML) enabled healthcare is now feasible for many health systems, yet little is known about effective strategies of system architecture and governance mechanisms for implementation. Our objective was to identify the different computational and organizational setups that early-adopter health systems have utilized to integrate AI/ML clinical decision support (AI-CDS) and scrutinize their trade-offs.

MATERIALS AND METHODS

We conducted structured interviews with health systems with AI deployment experience about their organizational and computational setups for deploying AI-CDS at point of care.

RESULTS

We contacted 34 health systems and interviewed 20 healthcare sites (58% response rate). Twelve (60%) sites used the native electronic health record vendor configuration for model development and deployment, making it the most common shared infrastructure. Nine (45%) sites used alternative computational configurations which varied significantly. Organizational configurations for managing AI-CDS were distinguished by how they identified model needs, built and implemented models, and were separable into 3 major types: Decentralized translation (n = 10, 50%), IT Department led (n = 2, 10%), and AI in Healthcare (AIHC) Team (n = 8, 40%).

DISCUSSION

No singular computational configuration enables all current use cases for AI-CDS. Health systems need to consider their desired applications for AI-CDS and whether investment in extending the off-the-shelf infrastructure is needed. Each organizational setup confers trade-offs for health systems planning strategies to implement AI-CDS.

CONCLUSION

Health systems will be able to use this framework to understand strengths and weaknesses of alternative organizational and computational setups when designing their strategy for artificial intelligence.

Exploring healthcare professionals' perceptions of artificial intelligence: Validating a questionnaire using the e-Delphi method

Objective

The aim of this study was to draw upon the collective knowledge of experts in the fields of health and technology to develop a questionnaire that measured healthcare professionals' perceptions of Artificial Intelligence (AI).

Methods

The panel for this study were carefully selected participants who demonstrated an interest and/or involvement in AI from the fields of health or information technology. Recruitment was accomplished via email which invited the panel member to participate and included study and consent information. Data were collected from three rounds in the form of an online survey, an online group meeting and email communication. A 75% median threshold was used to define consensus.

Results

Between January and March 2019, five healthcare professionals and three IT experts participated in three rounds of study to reach consensus on the structure and content of the questionnaire. In Round 1 panel members identified issues about general understanding of AI and achieved consensus on nine draft questionnaire items. In Round 2 the panel achieved consensus on demographic questions and comprehensive group discussion resulted in the development of two further questionnaire items for inclusion. In a final e-Delphi round, a draft of the final questionnaire was distributed via email to the panel members for comment. No further amendments were put forward and 100% consensus was achieved.

Conclusion

A modified e-Delphi method was used to validate and develop a questionnaire to explore healthcare professionals' perceptions of AI. The e-Delphi method was successful in achieving consensus from an interdisciplinary panel of experts from health and IT. Further research is recommended to test the reliability of this questionnaire.