1
|
Jokar M, Abdous A, Rahmanian V. AI chatbots in pet health care: Opportunities and challenges for owners. Vet Med Sci 2024; 10:e1464. [PMID: 38678576 PMCID: PMC11056198 DOI: 10.1002/vms3.1464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2024] [Accepted: 04/04/2024] [Indexed: 05/01/2024] Open
Abstract
The integration of artificial intelligence (AI) into health care has seen remarkable advancements, with applications extending to animal health. This article explores the potential benefits and challenges associated with employing AI chatbots as tools for pet health care. Focusing on ChatGPT, a prominent language model, the authors elucidate its capabilities and its potential impact on pet owners' decision-making processes. AI chatbots offer pet owners access to extensive information on animal health, research studies and diagnostic options, providing a cost-effective and convenient alternative to traditional veterinary consultations. The fate of a case involving a Border Collie named Sassy demonstrates the potential benefits of AI in veterinary medicine. In this instance, ChatGPT played a pivotal role in suggesting a diagnosis that led to successful treatment, showcasing the potential of AI chatbots as valuable tools in complex cases. However, concerns arise regarding pet owners relying solely on AI chatbots for medical advice, potentially resulting in misdiagnosis, inappropriate treatment and delayed professional intervention. We emphasize the need for a balanced approach, positioning AI chatbots as supplementary tools rather than substitutes for licensed veterinarians. To mitigate risks, the article proposes strategies such as educating pet owners on AI chatbots' limitations, implementing regulations to guide AI chatbot companies and fostering collaboration between AI chatbots and veterinarians. The intricate web of responsibilities in this dynamic landscape underscores the importance of government regulations, the educational role of AI chatbots and the symbiotic relationship between AI technology and veterinary expertise. In conclusion, while AI chatbots hold immense promise in transforming pet health care, cautious and informed usage is crucial. By promoting awareness, establishing regulations and fostering collaboration, the article advocates for a responsible integration of AI chatbots to ensure optimal care for pets.
Collapse
Affiliation(s)
- Mohammad Jokar
- Faculty of Veterinary MedicineKaraj BranchIslamic Azad UniversityKarajIran
| | - Arman Abdous
- Faculty of Veterinary MedicineKaraj BranchIslamic Azad UniversityKarajIran
| | - Vahid Rahmanian
- Department of Public HealthTorbat Jam Faculty of Medical SciencesTorbat JamIran
| |
Collapse
|
2
|
Ranjbar A, Mork EW, Ravn J, Brøgger H, Myrseth P, Østrem HP, Hallock H. Managing Risk and Quality of AI in Healthcare: Are Hospitals Ready for Implementation? Risk Manag Healthc Policy 2024; 17:877-882. [PMID: 38617593 PMCID: PMC11016246 DOI: 10.2147/rmhp.s452337] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Accepted: 03/20/2024] [Indexed: 04/16/2024] Open
Abstract
Artificial intelligence (AI) provides a unique opportunity to help meet the demands of the future healthcare system. However, hospitals may not be well equipped to handle safe and effective development and/or procurement of AI systems. Furthermore, upcoming regulations such as the EU AI Act may enforce the need to establish new management systems, quality assurance and control mechanisms, novel to healthcare organizations. This paper discusses challenges in AI implementation, particularly potential gaps in current management systems (MS), by reviewing the harmonized standard for AI MS, ISO 42001, as part of a gap analysis of a tertiary acute hospital with ongoing AI activities. Examination of the industry agnostic ISO 42001 reveals a technical debt within healthcare, aligning with previous research on digitalization and AI implementation. To successfully implement AI with quality assurance in mind, emphasis should be put on the foundation and structure of the healthcare organizations, including both workforce and data infrastructure.
Collapse
Affiliation(s)
- Arian Ranjbar
- Medical Technology and E-Health, Akershus University Hospital, Lørenskog, Norway
| | | | - Jesper Ravn
- Medical Technology and E-Health, Akershus University Hospital, Lørenskog, Norway
| | - Helga Brøgger
- Group Research and Development, DNV AS, Høvik, Norway
| | - Per Myrseth
- Group Research and Development, DNV AS, Høvik, Norway
| | | | - Harry Hallock
- Group Research and Development, DNV AS, Høvik, Norway
| |
Collapse
|
3
|
Moy S, Irannejad M, Manning SJ, Farahani M, Ahmed Y, Gao E, Prabhune R, Lorenz S, Mirza R, Klinger C. Patient Perspectives on the Use of Artificial Intelligence in Health Care: A Scoping Review. J Patient Cent Res Rev 2024; 11:51-62. [PMID: 38596349 PMCID: PMC11000703 DOI: 10.17294/2330-0698.2029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/11/2024] Open
Abstract
Purpose Artificial intelligence (AI) technology is being rapidly adopted into many different branches of medicine. Although research has started to highlight the impact of AI on health care, the focus on patient perspectives of AI is scarce. This scoping review aimed to explore the literature on adult patients' perspectives on the use of an array of AI technologies in the health care setting for design and deployment. Methods This scoping review followed Arksey and O'Malley's framework and Preferred Reporting Items for Systematic Reviews and Meta-Analysis for Scoping Reviews (PRISMA-ScR). To evaluate patient perspectives, we conducted a comprehensive literature search using eight interdisciplinary electronic databases, including grey literature. Articles published from 2015 to 2022 that focused on patient views regarding AI technology in health care were included. Thematic analysis was performed on the extracted articles. Results Of the 10,571 imported studies, 37 articles were included and extracted. From the 33 peer-reviewed and 4 grey literature articles, the following themes on AI emerged: (i) Patient attitudes, (ii) Influences on patient attitudes, (iii) Considerations for design, and (iv) Considerations for use. Conclusions Patients are key stakeholders essential to the uptake of AI in health care. The findings indicate that patients' needs and expectations are not fully considered in the application of AI in health care. Therefore, there is a need for patient voices in the development of AI in health care.
Collapse
Affiliation(s)
- Sally Moy
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Mona Irannejad
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | | | - Mehrdad Farahani
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Yomna Ahmed
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Ellis Gao
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Radhika Prabhune
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Suzan Lorenz
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Raza Mirza
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Christopher Klinger
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
- National Initiative for the Care of the Elderly, Toronto, Canada
| |
Collapse
|
4
|
Marco-Ruiz L, Hernández MÁT, Ngo PD, Makhlysheva A, Svenning TO, Dyb K, Chomutare T, Llatas CF, Muñoz-Gama J, Tayefi M. A multinational study on artificial intelligence adoption: Clinical implementers' perspectives. Int J Med Inform 2024; 184:105377. [PMID: 38377725 DOI: 10.1016/j.ijmedinf.2024.105377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 02/06/2024] [Accepted: 02/12/2024] [Indexed: 02/22/2024]
Abstract
BACKGROUND Despite substantial progress in AI research for healthcare, translating research achievements to AI systems in clinical settings is challenging and, in many cases, unsatisfactory. As a result, many AI investments have stalled at the prototype level, never reaching clinical settings. OBJECTIVE To improve the chances of future AI implementation projects succeeding, we analyzed the experiences of clinical AI system implementers to better understand the challenges and success factors in their implementations. METHODS Thirty-seven implementers of clinical AI from European and North and South American countries were interviewed. Semi-structured interviews were transcribed and analyzed qualitatively with the framework method, identifying the success factors and the reasons for challenges as well as documenting proposals from implementers to improve AI adoption in clinical settings. RESULTS We gathered the implementers' requirements for facilitating AI adoption in the clinical setting. The main findings include 1) the lesser importance of AI explainability in favor of proper clinical validation studies, 2) the need to actively involve clinical practitioners, and not only clinical researchers, in the inception of AI research projects, 3) the need for better information structures and processes to manage data access and the ethical approval of AI projects, 4) the need for better support for regulatory compliance and avoidance of duplications in data management approval bodies, 5) the need to increase both clinicians' and citizens' literacy as respects the benefits and limitations of AI, and 6) the need for better funding schemes to support the implementation, embedding, and validation of AI in the clinical workflow, beyond pilots. CONCLUSION Participants in the interviews are positive about the future of AI in clinical settings. At the same time, they proposenumerous measures to transfer research advancesinto implementations that will benefit healthcare personnel. Transferring AI research into benefits for healthcare workers and patients requires adjustments in regulations, data access procedures, education, funding schemes, and validation of AI systems.
Collapse
Affiliation(s)
- Luis Marco-Ruiz
- Norwegian Centre for E-Health Research, University Hospital of North Norway, Tromsø, Norway.
| | | | - Phuong Dinh Ngo
- Norwegian Centre for E-Health Research, University Hospital of North Norway, Tromsø, Norway
| | - Alexandra Makhlysheva
- Norwegian Centre for E-Health Research, University Hospital of North Norway, Tromsø, Norway
| | - Therese Olsen Svenning
- Norwegian Centre for E-Health Research, University Hospital of North Norway, Tromsø, Norway
| | - Kari Dyb
- Norwegian Centre for E-Health Research, University Hospital of North Norway, Tromsø, Norway
| | - Taridzo Chomutare
- Norwegian Centre for E-Health Research, University Hospital of North Norway, Tromsø, Norway
| | - Carlos Fernández Llatas
- Instituto de las Tecnologías de la Información y las Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Valencia, Spain
| | - Jorge Muñoz-Gama
- Department of Computer Science, Pontificia Universidad Católica de Chile, Santiago, Chile
| | - Maryam Tayefi
- Norwegian Centre for E-Health Research, University Hospital of North Norway, Tromsø, Norway
| |
Collapse
|
5
|
Al Hendi KD, Alyami MH, Alkahtany M, Dwivedi A, Alsaqour HG. Artificial intelligence in prosthodontics. Bioinformation 2024; 20:238-242. [PMID: 38712003 PMCID: PMC11069608 DOI: 10.6026/973206300200238] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Revised: 03/31/2024] [Accepted: 03/31/2024] [Indexed: 05/08/2024] Open
Abstract
Artificial Intelligence (AI) is gaining popularity worldwide owing to its significant impact in science and innovation. The role of AI in prosthodontics has increased significantly in recent years. AI is used for diagnosis, decision-making, prognosis, treatment planning and prediction of outcomes. Integration of AI into prosthodontics can enhance the accuracy and precision of dental practice. However, limited datasets are a major constraint in its practical applications.
Collapse
Affiliation(s)
| | | | - Mashary Alkahtany
- Prosthetic Dental Sciences, Faculty of Dentistry, Najran University, Najran, KSA
| | - Alok Dwivedi
- Prosthetic Dental Sciences, Faculty of Dentistry, Najran University, Najran, KSA
| | | |
Collapse
|
6
|
Trinkley KE, An R, Maw AM, Glasgow RE, Brownson RC. Leveraging artificial intelligence to advance implementation science: potential opportunities and cautions. Implement Sci 2024; 19:17. [PMID: 38383393 PMCID: PMC10880216 DOI: 10.1186/s13012-024-01346-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 01/25/2024] [Indexed: 02/23/2024] Open
Abstract
BACKGROUND The field of implementation science was developed to address the significant time delay between establishing an evidence-based practice and its widespread use. Although implementation science has contributed much toward bridging this gap, the evidence-to-practice chasm remains a challenge. There are some key aspects of implementation science in which advances are needed, including speed and assessing causality and mechanisms. The increasing availability of artificial intelligence applications offers opportunities to help address specific issues faced by the field of implementation science and expand its methods. MAIN TEXT This paper discusses the many ways artificial intelligence can address key challenges in applying implementation science methods while also considering potential pitfalls to the use of artificial intelligence. We answer the questions of "why" the field of implementation science should consider artificial intelligence, for "what" (the purpose and methods), and the "what" (consequences and challenges). We describe specific ways artificial intelligence can address implementation science challenges related to (1) speed, (2) sustainability, (3) equity, (4) generalizability, (5) assessing context and context-outcome relationships, and (6) assessing causality and mechanisms. Examples are provided from global health systems, public health, and precision health that illustrate both potential advantages and hazards of integrating artificial intelligence applications into implementation science methods. We conclude by providing recommendations and resources for implementation researchers and practitioners to leverage artificial intelligence in their work responsibly. CONCLUSIONS Artificial intelligence holds promise to advance implementation science methods ("why") and accelerate its goals of closing the evidence-to-practice gap ("purpose"). However, evaluation of artificial intelligence's potential unintended consequences must be considered and proactively monitored. Given the technical nature of artificial intelligence applications as well as their potential impact on the field, transdisciplinary collaboration is needed and may suggest the need for a subset of implementation scientists cross-trained in both fields to ensure artificial intelligence is used optimally and ethically.
Collapse
Affiliation(s)
- Katy E Trinkley
- Department of Family Medicine, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
- Adult and Child Center for Outcomes Research and Delivery Science Center, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
- Department of Biomedical Informatics, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
- Colorado Center for Personalized Medicine, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
| | - Ruopeng An
- Brown School and Division of Computational and Data Sciences at Washington University in St. Louis, St. Louis, MO, USA
| | - Anna M Maw
- Adult and Child Center for Outcomes Research and Delivery Science Center, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
- School of Medicine, Division of Hospital Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | - Russell E Glasgow
- Department of Family Medicine, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
- Adult and Child Center for Outcomes Research and Delivery Science Center, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | - Ross C Brownson
- Prevention Research Center, Brown School at Washington University in St. Louis, St. Louis, MO, USA
- Department of Surgery, Division of Public Health Sciences, and Alvin J. Siteman Cancer Center, Washington University School of Medicine, Washington University in St. Louis, St. Louis, MO, USA
| |
Collapse
|
7
|
Fisch U, Kliem P, Grzonka P, Sutter R. Performance of large language models on advocating the management of meningitis: a comparative qualitative study. BMJ Health Care Inform 2024; 31:e100978. [PMID: 38307617 PMCID: PMC10840049 DOI: 10.1136/bmjhci-2023-100978] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2023] [Accepted: 01/15/2024] [Indexed: 02/04/2024] Open
Abstract
OBJECTIVES We aimed to examine the adherence of large language models (LLMs) to bacterial meningitis guidelines using a hypothetical medical case, highlighting their utility and limitations in healthcare. METHODS A simulated clinical scenario of a patient with bacterial meningitis secondary to mastoiditis was presented in three independent sessions to seven publicly accessible LLMs (Bard, Bing, Claude-2, GTP-3.5, GTP-4, Llama, PaLM). Responses were evaluated for adherence to good clinical practice and two international meningitis guidelines. RESULTS A central nervous system infection was identified in 90% of LLM sessions. All recommended imaging, while 81% suggested lumbar puncture. Blood cultures and specific mastoiditis work-up were proposed in only 62% and 38% sessions, respectively. Only 38% of sessions provided the correct empirical antibiotic treatment, while antiviral treatment and dexamethasone were advised in 33% and 24%, respectively. Misleading statements were generated in 52%. No significant correlation was found between LLMs' text length and performance (r=0.29, p=0.20). Among all LLMs, GTP-4 demonstrated the best performance. DISCUSSION Latest LLMs provide valuable advice on differential diagnosis and diagnostic procedures but significantly vary in treatment-specific information for bacterial meningitis when introduced to a realistic clinical scenario. Misleading statements were common, with performance differences attributed to each LLM's unique algorithm rather than output length. CONCLUSIONS Users must be aware of such limitations and performance variability when considering LLMs as a support tool for medical decision-making. Further research is needed to refine these models' comprehension of complex medical scenarios and their ability to provide reliable information.
Collapse
Affiliation(s)
- Urs Fisch
- Department of Neurology, University Hospital Basel, Basel, Switzerland
| | - Paulina Kliem
- Clinic for Intensive Care Medicine, University Hospital Basel, Basel, Switzerland
| | - Pascale Grzonka
- Clinic for Intensive Care Medicine, University Hospital Basel, Basel, Switzerland
| | - Raoul Sutter
- Department of Neurology, University Hospital Basel, Basel, Switzerland
- Clinic for Intensive Care Medicine, University Hospital Basel, Basel, Switzerland
- Medical Faculty, University Basel, Basel, Switzerland
| |
Collapse
|
8
|
Rogan J, Bucci S, Firth J. Health Care Professionals' Views on the Use of Passive Sensing, AI, and Machine Learning in Mental Health Care: Systematic Review With Meta-Synthesis. JMIR Ment Health 2024; 11:e49577. [PMID: 38261403 PMCID: PMC10848143 DOI: 10.2196/49577] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Revised: 10/30/2023] [Accepted: 11/01/2023] [Indexed: 01/24/2024] Open
Abstract
BACKGROUND Mental health difficulties are highly prevalent worldwide. Passive sensing technologies and applied artificial intelligence (AI) methods can provide an innovative means of supporting the management of mental health problems and enhancing the quality of care. However, the views of stakeholders are important in understanding the potential barriers to and facilitators of their implementation. OBJECTIVE This study aims to review, critically appraise, and synthesize qualitative findings relating to the views of mental health care professionals on the use of passive sensing and AI in mental health care. METHODS A systematic search of qualitative studies was performed using 4 databases. A meta-synthesis approach was used, whereby studies were analyzed using an inductive thematic analysis approach within a critical realist epistemological framework. RESULTS Overall, 10 studies met the eligibility criteria. The 3 main themes were uses of passive sensing and AI in clinical practice, barriers to and facilitators of use in practice, and consequences for service users. A total of 5 subthemes were identified: barriers, facilitators, empowerment, risk to well-being, and data privacy and protection issues. CONCLUSIONS Although clinicians are open-minded about the use of passive sensing and AI in mental health care, important factors to consider are service user well-being, clinician workloads, and therapeutic relationships. Service users and clinicians must be involved in the development of digital technologies and systems to ensure ease of use. The development of, and training in, clear policies and guidelines on the use of passive sensing and AI in mental health care, including risk management and data security procedures, will also be key to facilitating clinician engagement. The means for clinicians and service users to provide feedback on how the use of passive sensing and AI in practice is being received should also be considered. TRIAL REGISTRATION PROSPERO International Prospective Register of Systematic Reviews CRD42022331698; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=331698.
Collapse
Affiliation(s)
- Jessica Rogan
- Division of Psychology and Mental Health, School of Health Sciences, Faculty of Biology, Medicine and Health, Manchester Academic Health Sciences, The University of Manchester, Manchester, United Kingdom
- Greater Manchester Mental Health NHS Foundation Trust, Manchester, United Kingdom
| | - Sandra Bucci
- Division of Psychology and Mental Health, School of Health Sciences, Faculty of Biology, Medicine and Health, Manchester Academic Health Sciences, The University of Manchester, Manchester, United Kingdom
- Greater Manchester Mental Health NHS Foundation Trust, Manchester, United Kingdom
| | - Joseph Firth
- Division of Psychology and Mental Health, School of Health Sciences, Faculty of Biology, Medicine and Health, Manchester Academic Health Sciences, The University of Manchester, Manchester, United Kingdom
| |
Collapse
|
9
|
van der Vegt AH, Campbell V, Mitchell I, Malycha J, Simpson J, Flenady T, Flabouris A, Lane PJ, Mehta N, Kalke VR, Decoyna JA, Es’haghi N, Liu CH, Scott IA. Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain. J Am Med Inform Assoc 2024; 31:509-524. [PMID: 37964688 PMCID: PMC10797271 DOI: 10.1093/jamia/ocad220] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Revised: 10/27/2023] [Accepted: 10/31/2023] [Indexed: 11/16/2023] Open
Abstract
OBJECTIVE To identify factors influencing implementation of machine learning algorithms (MLAs) that predict clinical deterioration in hospitalized adult patients and relate these to a validated implementation framework. MATERIALS AND METHODS A systematic review of studies of implemented or trialed real-time clinical deterioration prediction MLAs was undertaken, which identified: how MLA implementation was measured; impact of MLAs on clinical processes and patient outcomes; and barriers, enablers and uncertainties within the implementation process. Review findings were then mapped to the SALIENT end-to-end implementation framework to identify the implementation stages at which these factors applied. RESULTS Thirty-seven articles relating to 14 groups of MLAs were identified, each trialing or implementing a bespoke algorithm. One hundred and seven distinct implementation evaluation metrics were identified. Four groups reported decreased hospital mortality, 1 significantly. We identified 24 barriers, 40 enablers, and 14 uncertainties and mapped these to the 5 stages of the SALIENT implementation framework. DISCUSSION Algorithm performance across implementation stages decreased between in silico and trial stages. Silent plus pilot trial inclusion was associated with decreased mortality, as was the use of logistic regression algorithms that used less than 39 variables. Mitigation of alert fatigue via alert suppression and threshold configuration was commonly employed across groups. CONCLUSIONS : There is evidence that real-world implementation of clinical deterioration prediction MLAs may improve clinical outcomes. Various factors identified as influencing success or failure of implementation can be mapped to different stages of implementation, thereby providing useful and practical guidance for implementers.
Collapse
Affiliation(s)
- Anton H van der Vegt
- Centre for Health Services Research, The University of Queensland, Brisbane, QLD 4102, Australia
| | - Victoria Campbell
- Intensive Care Unit, Sunshine Coast Hospital and Health Service, Birtynia, QLD 4575, Australia
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Imogen Mitchell
- Office of Research and Education, Canberra Health Services, Canberra, ACT 2601, Australia
| | - James Malycha
- Department of Critical Care Medicine, The Queen Elizabeth Hospital, Woodville, SA 5011, Australia
| | - Joanna Simpson
- Eastern Health Intensive Care Services, Eastern Health, Box Hill, VIC 3128, Australia
| | - Tracy Flenady
- School of Nursing, Midwifery & Social Sciences, Central Queensland University, Rockhampton, QLD 4701, Australia
| | - Arthas Flabouris
- Intensive Care Department, Royal Adelaide Hospital, Adelaide, SA 5000, Australia
- Adelaide Medical School, University of Adelaide, Adelaide, SA 5005, Australia
| | - Paul J Lane
- Safety Quality & Innovation, The Prince Charles Hospital, Chermside, QLD 4032, Australia
| | - Naitik Mehta
- Patient Safety and Quality, Clinical Excellence Queensland, Brisbane, QLD 4001, Australia
| | - Vikrant R Kalke
- Patient Safety and Quality, Clinical Excellence Queensland, Brisbane, QLD 4001, Australia
| | - Jovie A Decoyna
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Nicholas Es’haghi
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Chun-Huei Liu
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Ian A Scott
- Centre for Health Services Research, The University of Queensland, Brisbane, QLD 4102, Australia
- Department of Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Brisbane, QLD 4102, Australia
| |
Collapse
|
10
|
Nilsen P, Svedberg P, Neher M, Nair M, Larsson I, Petersson L, Nygren J. A Framework to Guide Implementation of AI in Health Care: Protocol for a Cocreation Research Project. JMIR Res Protoc 2023; 12:e50216. [PMID: 37938896 PMCID: PMC10666006 DOI: 10.2196/50216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Revised: 08/16/2023] [Accepted: 09/08/2023] [Indexed: 11/10/2023] Open
Abstract
BACKGROUND Artificial intelligence (AI) has the potential in health care to transform patient care and administrative processes, yet health care has been slow to adopt AI due to many types of barriers. Implementation science has shown the importance of structured implementation processes to overcome implementation barriers. However, there is a lack of knowledge and tools to guide such processes when implementing AI-based applications in health care. OBJECTIVE The aim of this protocol is to describe the development, testing, and evaluation of a framework, "Artificial Intelligence-Quality Implementation Framework" (AI-QIF), intended to guide decisions and activities related to the implementation of various AI-based applications in health care. METHODS The paper outlines the development of an AI implementation framework for broad use in health care based on the Quality Implementation Framework (QIF). QIF is a process model developed in implementation science. The model guides the user to consider implementation-related issues in a step-by-step design and plan and perform activities that support implementation. This framework was chosen for its adaptability, usability, broad scope, and detailed guidance concerning important activities and considerations for successful implementation. The development will proceed in 5 phases with primarily qualitative methods being used. The process starts with phase I, in which an AI-adapted version of QIF is created (AI-QIF). Phase II will produce a digital mockup of the AI-QIF. Phase III will involve the development of a prototype of the AI-QIF with an intuitive user interface. Phase IV is dedicated to usability testing of the prototype in health care environments. Phase V will focus on evaluating the usability and effectiveness of the AI-QIF. Cocreation is a guiding principle for the project and is an important aspect in 4 of the 5 development phases. The cocreation process will enable the use of both on research-based and practice-based knowledge. RESULTS The project is being conducted within the frame of a larger research program, with the overall objective of developing theoretically and empirically informed frameworks to support AI implementation in routine health care. The program was launched in 2021 and has carried out numerous research activities. The development of AI-QIF as a tool to guide the implementation of AI-based applications in health care will draw on knowledge and experience acquired from these activities. The framework is being developed over 2 years, from January 2023 to December 2024. It is under continuous development and refinement. CONCLUSIONS The development of the AI implementation framework, AI-QIF, described in this study protocol aims to facilitate the implementation of AI-based applications in health care based on the premise that implementation processes benefit from being well-prepared and structured. The framework will be coproduced to enhance its relevance, validity, usefulness, and potential value for application in practice. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/50216.
Collapse
Affiliation(s)
- Per Nilsen
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden
| | - Petra Svedberg
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Margit Neher
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Monika Nair
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Ingrid Larsson
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Lena Petersson
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Jens Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|
11
|
Fernando M, Abell B, Tyack Z, Donovan T, McPhail SM, Naicker S. Using Theories, Models, and Frameworks to Inform Implementation Cycles of Computerized Clinical Decision Support Systems in Tertiary Health Care Settings: Scoping Review. J Med Internet Res 2023; 25:e45163. [PMID: 37851492 PMCID: PMC10620641 DOI: 10.2196/45163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2022] [Revised: 08/18/2023] [Accepted: 09/14/2023] [Indexed: 10/19/2023] Open
Abstract
BACKGROUND Computerized clinical decision support systems (CDSSs) are essential components of modern health system service delivery, particularly within acute care settings such as hospitals. Theories, models, and frameworks may assist in facilitating the implementation processes associated with CDSS innovation and its use within these care settings. These processes include context assessments to identify key determinants, implementation plans for adoption, promoting ongoing uptake, adherence, and long-term evaluation. However, there has been no prior review synthesizing the literature regarding the theories, models, and frameworks that have informed the implementation and adoption of CDSSs within hospitals. OBJECTIVE This scoping review aims to identify the theory, model, and framework approaches that have been used to facilitate the implementation and adoption of CDSSs in tertiary health care settings, including hospitals. The rationales reported for selecting these approaches, including the limitations and strengths, are described. METHODS A total of 5 electronic databases were searched (CINAHL via EBSCOhost, PubMed, Scopus, PsycINFO, and Embase) to identify studies that implemented or adopted a CDSS in a tertiary health care setting using an implementation theory, model, or framework. No date or language limits were applied. A narrative synthesis was conducted using full-text publications and abstracts. Implementation phases were classified according to the "Active Implementation Framework stages": exploration (feasibility and organizational readiness), installation (organizational preparation), initial implementation (initiating implementation, ie, training), full implementation (sustainment), and nontranslational effectiveness studies. RESULTS A total of 81 records (42 full text and 39 abstracts) were included. Full-text studies and abstracts are reported separately. For full-text studies, models (18/42, 43%), followed by determinants frameworks (14/42,33%), were most frequently used to guide adoption and evaluation strategies. Most studies (36/42, 86%) did not list the limitations associated with applying a specific theory, model, or framework. CONCLUSIONS Models and related quality improvement methods were most frequently used to inform CDSS adoption. Models were not typically combined with each other or with theory to inform full-cycle implementation strategies. The findings highlight a gap in the application of implementation methods including theories, models, and frameworks to facilitate full-cycle implementation strategies for hospital CDSSs.
Collapse
Affiliation(s)
- Manasha Fernando
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, Australia
| | - Bridget Abell
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, Australia
| | - Zephanie Tyack
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, Australia
| | - Thomasina Donovan
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, Australia
| | - Steven M McPhail
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, Australia
- Digital Health and Informatics Directorate, Metro South Health, Brisbane, Australia
| | - Sundresan Naicker
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, Australia
| |
Collapse
|
12
|
Cresswell K, Rigby M, Magrabi F, Scott P, Brender J, Craven CK, Wong ZSY, Kukhareva P, Ammenwerth E, Georgiou A, Medlock S, De Keizer NF, Nykänen P, Prgomet M, Williams R. The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision. Health Policy 2023; 136:104889. [PMID: 37579545 DOI: 10.1016/j.healthpol.2023.104889] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 08/04/2023] [Indexed: 08/16/2023]
Abstract
Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.
Collapse
Affiliation(s)
- Kathrin Cresswell
- The University of Edinburgh, Usher Institute, Edinburgh, United Kingdom.
| | - Michael Rigby
- Keele University, School of Social, Political and Global Studies and School of Primary, Community and Social Care, Keele, United Kingdom
| | - Farah Magrabi
- Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
| | - Philip Scott
- University of Wales Trinity Saint David, Swansea, United Kingdom
| | - Jytte Brender
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Catherine K Craven
- University of Texas Health Science Center at San Antonio, San Antonio, TX, United States
| | - Zoie Shui-Yee Wong
- St. Luke's International University, Graduate School of Public Health, Tokyo, Japan
| | - Polina Kukhareva
- Department of Biomedical Informatics, University of Utah, United States of America
| | - Elske Ammenwerth
- UMIT TIROL, Private University for Health Sciences and Health Informatics, Institute of Medical Informatics, Hall in Tirol, Austria
| | - Andrew Georgiou
- Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
| | - Stephanie Medlock
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Digital Health and Quality of Care Amsterdam, the Netherlands
| | - Nicolette F De Keizer
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Digital Health and Quality of Care Amsterdam, the Netherlands
| | - Pirkko Nykänen
- Tampere University, Faculty for Information Technology and Communication Sciences, Finland
| | - Mirela Prgomet
- Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, Australia
| | - Robin Williams
- The University of Edinburgh, Institute for the Study of Science, Technology and Innovation, Edinburgh, United Kingdom
| |
Collapse
|
13
|
El Naqa I, Karolak A, Luo Y, Folio L, Tarhini AA, Rollison D, Parodi K. Translation of AI into oncology clinical practice. Oncogene 2023; 42:3089-3097. [PMID: 37684407 DOI: 10.1038/s41388-023-02826-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2023] [Revised: 08/23/2023] [Accepted: 08/25/2023] [Indexed: 09/10/2023]
Abstract
Artificial intelligence (AI) is a transformative technology that is capturing popular imagination and can revolutionize biomedicine. AI and machine learning (ML) algorithms have the potential to break through existing barriers in oncology research and practice such as automating workflow processes, personalizing care, and reducing healthcare disparities. Emerging applications of AI/ML in the literature include screening and early detection of cancer, disease diagnosis, response prediction, prognosis, and accelerated drug discovery. Despite this excitement, only few AI/ML models have been properly validated and fewer have become regulated products for routine clinical use. In this review, we highlight the main challenges impeding AI/ML clinical translation. We present different clinical use cases from the domains of radiology, radiation oncology, immunotherapy, and drug discovery in oncology. We dissect the unique challenges and opportunities associated with each of these cases. Finally, we summarize the general requirements for successful AI/ML implementation in the clinic, highlighting specific examples and points of emphasis including the importance of multidisciplinary collaboration of stakeholders, role of domain experts in AI augmentation, transparency of AI/ML models, and the establishment of a comprehensive quality assurance program to mitigate risks of training bias and data drifts, all culminating toward safer and more beneficial AI/ML applications in oncology labs and clinics.
Collapse
Affiliation(s)
- Issam El Naqa
- Department of Machine Learning, Moffitt Cancer Center, Tampa, FL, 33612, USA.
| | - Aleksandra Karolak
- Department of Machine Learning, Moffitt Cancer Center, Tampa, FL, 33612, USA
| | - Yi Luo
- Department of Machine Learning, Moffitt Cancer Center, Tampa, FL, 33612, USA
| | - Les Folio
- Diagnostic Imaging & Interventional Radiology, Moffitt Cancer Center, Tampa, FL, 33612, USA
| | - Ahmad A Tarhini
- Cutaneous Oncology and Immunology, Moffitt Cancer Center, Tampa, FL, 33612, USA
| | - Dana Rollison
- Department of Cancer Epidemiology, Moffitt Cancer Center, Tampa, FL, 33612, USA
| | - Katia Parodi
- Department of Medical Physics, Ludwig-Maximilians-Universität München, Munich, Germany
| |
Collapse
|
14
|
Ardito V, Cappellaro G, Compagni A, Petracca F, Preti LM. Implementation of Machine Learning Applications in Health Care Organizations: Protocol for a Systematic Review of Empirical Studies. JMIR Res Protoc 2023; 12:e47971. [PMID: 37698910 PMCID: PMC10523208 DOI: 10.2196/47971] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Revised: 06/11/2023] [Accepted: 07/17/2023] [Indexed: 09/13/2023] Open
Abstract
BACKGROUND An increasing interest in machine learning (ML) has been observed among scholars and health care professionals. However, while ML-based applications have been shown to be effective and have the potential to change the delivery of patient care, their implementation in health care organizations is complex. There are several challenges that currently hamper the uptake of ML in daily practice, and there is currently limited knowledge on how these challenges have been addressed in empirical studies on implemented ML-based applications. OBJECTIVE The aim of this systematic literature review is twofold: (1) to map the ML-based applications implemented in health care organizations, with a focus on investigating the organizational dimensions that are relevant in the implementation process; and (2) to analyze the processes and strategies adopted to foster a successful uptake of ML. METHODS We developed this protocol following the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) guidelines. The search was conducted on 3 databases (PubMed, Scopus, and Web of Science), considering a 10-year time frame (2013-2023). The search strategy was built around 4 blocks of keywords (artificial intelligence, implementation, health care, and study type). Based on the detailed inclusion criteria defined, only empirical studies documenting the implementation of ML-based applications used by health care professionals in clinical settings will be considered. The study protocol was registered in PROSPERO (International Prospective Register of Systematic Reviews). RESULTS The review is ongoing and is expected to be completed by September 2023. Data analysis is currently underway, and the first results are expected to be submitted for publication in November 2023. The study was funded by the European Union within the Multilayered Urban Sustainability Action (MUSA) project. CONCLUSIONS ML-based applications involving clinical decision support and automation of clinical tasks present unique traits that add several layers of complexity compared with earlier health technologies. Our review aims at contributing to the existing literature by investigating the implementation of ML from an organizational perspective and by systematizing a conspicuous amount of information on factors influencing implementation. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/47971.
Collapse
Affiliation(s)
- Vittoria Ardito
- Center for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy
| | - Giulia Cappellaro
- Center for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy
- Department of Social and Political Sciences, Bocconi University, Milan, Italy
| | - Amelia Compagni
- Center for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy
- Department of Social and Political Sciences, Bocconi University, Milan, Italy
| | - Francesco Petracca
- Center for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy
| | - Luigi Maria Preti
- Center for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy
| |
Collapse
|
15
|
Steerling E, Siira E, Nilsen P, Svedberg P, Nygren J. Implementing AI in healthcare-the relevance of trust: a scoping review. FRONTIERS IN HEALTH SERVICES 2023; 3:1211150. [PMID: 37693234 PMCID: PMC10484529 DOI: 10.3389/frhs.2023.1211150] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 08/11/2023] [Indexed: 09/12/2023]
Abstract
Background The process of translation of AI and its potential benefits into practice in healthcare services has been slow in spite of its rapid development. Trust in AI in relation to implementation processes is an important aspect. Without a clear understanding, the development of effective implementation strategies will not be possible, nor will AI advance despite the significant investments and possibilities. Objective This study aimed to explore the scientific literature regarding how trust in AI in relation to implementation in healthcare is conceptualized and what influences trust in AI in relation to implementation in healthcare. Methods This scoping review included five scientific databases. These were searched to identify publications related to the study aims. Articles were included if they were published in English, after 2012, and peer-reviewed. Two independent reviewers conducted an abstract and full-text review, as well as carrying out a thematic analysis with an inductive approach to address the study aims. The review was reported in accordance with the PRISMA-ScR guidelines. Results A total of eight studies were included in the final review. We found that trust was conceptualized in different ways. Most empirical studies had an individual perspective where trust was directed toward the technology's capability. Two studies focused on trust as relational between people in the context of the AI application rather than as having trust in the technology itself. Trust was also understood by its determinants and as having a mediating role, positioned between characteristics and AI use. The thematic analysis yielded three themes: individual characteristics, AI characteristics and contextual characteristics, which influence trust in AI in relation to implementation in healthcare. Conclusions Findings showed that the conceptualization of trust in AI differed between the studies, as well as which determinants they accounted for as influencing trust. Few studies looked beyond individual characteristics and AI characteristics. Future empirical research addressing trust in AI in relation to implementation in healthcare should have a more holistic view of the concept to be able to manage the many challenges, uncertainties, and perceived risks.
Collapse
Affiliation(s)
- Emilie Steerling
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Elin Siira
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Per Nilsen
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden
| | - Petra Svedberg
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Jens Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|
16
|
Nair M, Andersson J, Nygren JM, Lundgren LE. Barriers and Enablers for Implementation of an Artificial Intelligence-Based Decision Support Tool to Reduce the Risk of Readmission of Patients With Heart Failure: Stakeholder Interviews. JMIR Form Res 2023; 7:e47335. [PMID: 37610799 PMCID: PMC10483295 DOI: 10.2196/47335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Revised: 05/29/2023] [Accepted: 05/31/2023] [Indexed: 08/24/2023] Open
Abstract
BACKGROUND Artificial intelligence (AI) applications in health care are expected to provide value for health care organizations, professionals, and patients. However, the implementation of such systems should be carefully planned and organized in order to ensure quality, safety, and acceptance. The gathered view of different stakeholders is a great source of information to understand the barriers and enablers for implementation in a specific context. OBJECTIVE This study aimed to understand the context and stakeholder perspectives related to the future implementation of a clinical decision support system for predicting readmissions of patients with heart failure. The study was part of a larger project involving model development, interface design, and implementation planning of the system. METHODS Interviews were held with 12 stakeholders from the regional and municipal health care organizations to gather their views on the potential effects implementation of such a decision support system could have as well as barriers and enablers for implementation. Data were analyzed based on the categories defined in the nonadoption, abandonment, scale-up, spread, sustainability (NASSS) framework. RESULTS Stakeholders had in general a positive attitude and curiosity toward AI-based decision support systems, and mentioned several barriers and enablers based on the experiences of previous implementations of information technology systems. Central aspects to consider for the proposed clinical decision support system were design aspects, access to information throughout the care process, and integration into the clinical workflow. The implementation of such a system could lead to a number of effects related to both clinical outcomes as well as resource allocation, which are all important to address in the planning of implementation. Stakeholders saw, however, value in several aspects of implementing such system, emphasizing the increased quality of life for those patients who can avoid being hospitalized. CONCLUSIONS Several ideas were put forward on how the proposed AI system would potentially affect and provide value for patients, professionals, and the organization, and implementation aspects were important parts of that. A successful system can help clinicians to prioritize the need for different types of treatments but also be used for planning purposes within the hospital. However, the system needs not only technological and clinical precision but also a carefully planned implementation process. Such a process should take into consideration the aspects related to all the categories in the NASSS framework. This study further highlighted the importance to study stakeholder needs early in the process of development, design, and implementation of decision support systems, as the data revealed new information on the potential use of the system and the placement of the application in the care process.
Collapse
Affiliation(s)
- Monika Nair
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | | | - Jens M Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Lina E Lundgren
- School of Business, Innovation and Sustainability, Halmstad University, Halmstad, Sweden
| |
Collapse
|
17
|
van der Vegt AH, Scott IA, Dermawan K, Schnetler RJ, Kalke VR, Lane PJ. Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework. J Am Med Inform Assoc 2023; 30:1503-1515. [PMID: 37208863 PMCID: PMC10436156 DOI: 10.1093/jamia/ocad088] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 04/17/2023] [Accepted: 05/09/2023] [Indexed: 05/21/2023] Open
Abstract
OBJECTIVE To derive a comprehensive implementation framework for clinical AI models within hospitals informed by existing AI frameworks and integrated with reporting standards for clinical AI research. MATERIALS AND METHODS (1) Derive a provisional implementation framework based on the taxonomy of Stead et al and integrated with current reporting standards for AI research: TRIPOD, DECIDE-AI, CONSORT-AI. (2) Undertake a scoping review of published clinical AI implementation frameworks and identify key themes and stages. (3) Perform a gap analysis and refine the framework by incorporating missing items. RESULTS The provisional AI implementation framework, called SALIENT, was mapped to 5 stages common to both the taxonomy and the reporting standards. A scoping review retrieved 20 studies and 247 themes, stages, and subelements were identified. A gap analysis identified 5 new cross-stage themes and 16 new tasks. The final framework comprised 5 stages, 7 elements, and 4 components, including the AI system, data pipeline, human-computer interface, and clinical workflow. DISCUSSION This pragmatic framework resolves gaps in existing stage- and theme-based clinical AI implementation guidance by comprehensively addressing the what (components), when (stages), and how (tasks) of AI implementation, as well as the who (organization) and why (policy domains). By integrating research reporting standards into SALIENT, the framework is grounded in rigorous evaluation methodologies. The framework requires validation as being applicable to real-world studies of deployed AI models. CONCLUSIONS A novel end-to-end framework has been developed for implementing AI within hospital clinical practice that builds on previous AI implementation frameworks and research reporting standards.
Collapse
Affiliation(s)
- Anton H van der Vegt
- Centre for Health Services Research, The University of Queensland, Brisbane, Australia
| | - Ian A Scott
- Department of Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Brisbane, Australia
| | - Krishna Dermawan
- Centre for Information Resilience, The University of Queensland, St Lucia, Australia
| | - Rudolf J Schnetler
- School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia, Australia
| | - Vikrant R Kalke
- Patient Safety and Quality, Clinical Excellence Queensland, Queensland Health, Brisbane, Australia
| | - Paul J Lane
- Safety Quality & Innovation, The Prince Charles Hospital, Queensland Health, Brisbane, Australia
| |
Collapse
|
18
|
Abell B, Naicker S, Rodwell D, Donovan T, Tariq A, Baysari M, Blythe R, Parsons R, McPhail SM. Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review. Implement Sci 2023; 18:32. [PMID: 37495997 PMCID: PMC10373265 DOI: 10.1186/s13012-023-01287-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2023] [Accepted: 07/17/2023] [Indexed: 07/28/2023] Open
Abstract
BACKGROUND Successful implementation and utilization of Computerized Clinical Decision Support Systems (CDSS) in hospitals is complex and challenging. Implementation science, and in particular the Nonadoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) framework, may offer a systematic approach for identifying and addressing these challenges. This review aimed to identify, categorize, and describe barriers and facilitators to CDSS implementation in hospital settings and map them to the NASSS framework. Exploring the applicability of the NASSS framework to CDSS implementation was a secondary aim. METHODS Electronic database searches were conducted (21 July 2020; updated 5 April 2022) in Ovid MEDLINE, Embase, Scopus, PyscInfo, and CINAHL. Original research studies reporting on measured or perceived barriers and/or facilitators to implementation and adoption of CDSS in hospital settings, or attitudes of healthcare professionals towards CDSS were included. Articles with a primary focus on CDSS development were excluded. No language or date restrictions were applied. We used qualitative content analysis to identify determinants and organize them into higher-order themes, which were then reflexively mapped to the NASSS framework. RESULTS Forty-four publications were included. These comprised a range of study designs, geographic locations, participants, technology types, CDSS functions, and clinical contexts of implementation. A total of 227 individual barriers and 130 individual facilitators were identified across the included studies. The most commonly reported influences on implementation were fit of CDSS with workflows (19 studies), the usefulness of the CDSS output in practice (17 studies), CDSS technical dependencies and design (16 studies), trust of users in the CDSS input data and evidence base (15 studies), and the contextual fit of the CDSS with the user's role or clinical setting (14 studies). Most determinants could be appropriately categorized into domains of the NASSS framework with barriers and facilitators in the "Technology," "Organization," and "Adopters" domains most frequently reported. No determinants were assigned to the "Embedding and Adaptation Over Time" domain. CONCLUSIONS This review identified the most common determinants which could be targeted for modification to either remove barriers or facilitate the adoption and use of CDSS within hospitals. Greater adoption of implementation theory should be encouraged to support CDSS implementation.
Collapse
Affiliation(s)
- Bridget Abell
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia
| | - Sundresan Naicker
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia.
| | - David Rodwell
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia
| | - Thomasina Donovan
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia
| | - Amina Tariq
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia
| | - Melissa Baysari
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Camperdown, Australia
| | - Robin Blythe
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia
| | - Rex Parsons
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia
| | - Steven M McPhail
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia
| |
Collapse
|
19
|
Neher M, Petersson L, Nygren JM, Svedberg P, Larsson I, Nilsen P. Innovation in healthcare: leadership perceptions about the innovation characteristics of artificial intelligence-a qualitative interview study with healthcare leaders in Sweden. Implement Sci Commun 2023; 4:81. [PMID: 37464420 DOI: 10.1186/s43058-023-00458-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2022] [Accepted: 06/17/2023] [Indexed: 07/20/2023] Open
Abstract
BACKGROUND Despite the extensive hopes and expectations for value creation resulting from the implementation of artificial intelligence (AI) applications in healthcare, research has predominantly been technology-centric rather than focused on the many changes that are required in clinical practice for the technology to be successfully implemented. The importance of leaders in the successful implementation of innovations in healthcare is well recognised, yet their perspectives on the specific innovation characteristics of AI are still unknown. The aim of this study was therefore to explore the perceptions of leaders in healthcare concerning the innovation characteristics of AI intended to be implemented into their organisation. METHODS The study had a deductive qualitative design, using constructs from the innovation domain in the Consolidated Framework for Implementation Research (CFIR). Interviews were conducted with 26 leaders in healthcare. RESULTS Participants perceived that AI could provide relative advantages when it came to care management, supporting clinical decisions, and the early detection of disease and risk of disease. The development of AI in the organisation itself was perceived as the main current innovation source. The evidence base behind AI technology was questioned, in relation to its transparency, potential quality improvement, and safety risks. Although the participants acknowledged AI to be superior to human action in terms of effectiveness and precision in some situations, they also expressed uncertainty about the adaptability and trialability of AI. Complexities such as the characteristics of the technology, the lack of conceptual consensus about AI, and the need for a variety of implementation strategies to accomplish transformative change in practice were identified, as were uncertainties about the costs involved in AI implementation. CONCLUSION Healthcare leaders not only saw potential in the technology and its use in practice, but also felt that AI's opacity limits its evidence strength and that complexities in relation to AI itself and its implementation influence its current use in healthcare practice. More research is needed based on actual experiences using AI applications in real-world situations and their impact on clinical practice. New theories, models, and frameworks may need to be developed to meet challenges related to the implementation of AI in healthcare.
Collapse
Affiliation(s)
- Margit Neher
- School of Health and Welfare, Halmstad University, Box 823, SE-30118, Halmstad, Sweden.
| | - Lena Petersson
- School of Health and Welfare, Halmstad University, Box 823, SE-30118, Halmstad, Sweden
| | - Jens M Nygren
- School of Health and Welfare, Halmstad University, Box 823, SE-30118, Halmstad, Sweden
| | - Petra Svedberg
- School of Health and Welfare, Halmstad University, Box 823, SE-30118, Halmstad, Sweden
| | - Ingrid Larsson
- School of Health and Welfare, Halmstad University, Box 823, SE-30118, Halmstad, Sweden
| | - Per Nilsen
- School of Health and Welfare, Halmstad University, Box 823, SE-30118, Halmstad, Sweden
- Department of Health, Medicine and Caring Sciences, Division of Public Health, Faculty of Health Sciences, Linköping University, Linköping, Sweden
| |
Collapse
|
20
|
Brereton TA, Malik MM, Lifson M, Greenwood JD, Peterson KJ, Overgaard SM. The Role of Artificial Intelligence Model Documentation in Translational Science: Scoping Review. Interact J Med Res 2023; 12:e45903. [PMID: 37450330 PMCID: PMC10382950 DOI: 10.2196/45903] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2023] [Revised: 05/10/2023] [Accepted: 05/11/2023] [Indexed: 07/18/2023] Open
Abstract
BACKGROUND Despite the touted potential of artificial intelligence (AI) and machine learning (ML) to revolutionize health care, clinical decision support tools, herein referred to as medical modeling software (MMS), have yet to realize the anticipated benefits. One proposed obstacle is the acknowledged gaps in AI translation. These gaps stem partly from the fragmentation of processes and resources to support MMS transparent documentation. Consequently, the absence of transparent reporting hinders the provision of evidence to support the implementation of MMS in clinical practice, thereby serving as a substantial barrier to the successful translation of software from research settings to clinical practice. OBJECTIVE This study aimed to scope the current landscape of AI- and ML-based MMS documentation practices and elucidate the function of documentation in facilitating the translation of ethical and explainable MMS into clinical workflows. METHODS A scoping review was conducted in accordance with PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. PubMed was searched using Medical Subject Headings key concepts of AI, ML, ethical considerations, and explainability to identify publications detailing AI- and ML-based MMS documentation, in addition to snowball sampling of selected reference lists. To include the possibility of implicit documentation practices not explicitly labeled as such, we did not use documentation as a key concept but as an inclusion criterion. A 2-stage screening process (title and abstract screening and full-text review) was conducted by 1 author. A data extraction template was used to record publication-related information; barriers to developing ethical and explainable MMS; available standards, regulations, frameworks, or governance strategies related to documentation; and recommendations for documentation for papers that met the inclusion criteria. RESULTS Of the 115 papers retrieved, 21 (18.3%) papers met the requirements for inclusion. Ethics and explainability were investigated in the context of AI- and ML-based MMS documentation and translation. Data detailing the current state and challenges and recommendations for future studies were synthesized. Notable themes defining the current state and challenges that required thorough review included bias, accountability, governance, and explainability. Recommendations identified in the literature to address present barriers call for a proactive evaluation of MMS, multidisciplinary collaboration, adherence to investigation and validation protocols, transparency and traceability requirements, and guiding standards and frameworks that enhance documentation efforts and support the translation of AI- and ML-based MMS. CONCLUSIONS Resolving barriers to translation is critical for MMS to deliver on expectations, including those barriers identified in this scoping review related to bias, accountability, governance, and explainability. Our findings suggest that transparent strategic documentation, aligning translational science and regulatory science, will support the translation of MMS by coordinating communication and reporting and reducing translational barriers, thereby furthering the adoption of MMS.
Collapse
Affiliation(s)
- Tracey A Brereton
- Center for Digital Health, Mayo Clinic, Rochester, MN, United States
| | - Momin M Malik
- Center for Digital Health, Mayo Clinic, Rochester, MN, United States
| | - Mark Lifson
- Center for Digital Health, Mayo Clinic, Rochester, MN, United States
| | - Jason D Greenwood
- Department of Family Medicine, Mayo Clinic, Rochester, MN, United States
| | - Kevin J Peterson
- Center for Digital Health, Mayo Clinic, Rochester, MN, United States
| | | |
Collapse
|
21
|
Al Kuwaiti A, Nazer K, Al-Reedy A, Al-Shehri S, Al-Muhanna A, Subbarayalu AV, Al Muhanna D, Al-Muhanna FA. A Review of the Role of Artificial Intelligence in Healthcare. J Pers Med 2023; 13:951. [PMID: 37373940 PMCID: PMC10301994 DOI: 10.3390/jpm13060951] [Citation(s) in RCA: 19] [Impact Index Per Article: 19.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Revised: 05/11/2023] [Accepted: 05/12/2023] [Indexed: 06/29/2023] Open
Abstract
Artificial intelligence (AI) applications have transformed healthcare. This study is based on a general literature review uncovering the role of AI in healthcare and focuses on the following key aspects: (i) medical imaging and diagnostics, (ii) virtual patient care, (iii) medical research and drug discovery, (iv) patient engagement and compliance, (v) rehabilitation, and (vi) other administrative applications. The impact of AI is observed in detecting clinical conditions in medical imaging and diagnostic services, controlling the outbreak of coronavirus disease 2019 (COVID-19) with early diagnosis, providing virtual patient care using AI-powered tools, managing electronic health records, augmenting patient engagement and compliance with the treatment plan, reducing the administrative workload of healthcare professionals (HCPs), discovering new drugs and vaccines, spotting medical prescription errors, extensive data storage and analysis, and technology-assisted rehabilitation. Nevertheless, this science pitch meets several technical, ethical, and social challenges, including privacy, safety, the right to decide and try, costs, information and consent, access, and efficacy, while integrating AI into healthcare. The governance of AI applications is crucial for patient safety and accountability and for raising HCPs' belief in enhancing acceptance and boosting significant health consequences. Effective governance is a prerequisite to precisely address regulatory, ethical, and trust issues while advancing the acceptance and implementation of AI. Since COVID-19 hit the global health system, the concept of AI has created a revolution in healthcare, and such an uprising could be another step forward to meet future healthcare needs.
Collapse
Affiliation(s)
- Ahmed Al Kuwaiti
- Department of Dental Education, College of Dentistry, Deanship of Quality and Academic Accreditation, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
| | - Khalid Nazer
- Department of Information and Technology, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
- Health Information Department, King Fahad hospital of the University, Al-Khobar 31952, Saudi Arabia
| | - Abdullah Al-Reedy
- Department of Information and Technology, Family and Community Medicine Department, Family and Community Medicine Centre, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
| | - Shaher Al-Shehri
- Faculty of Medicine, Family and Community Medicine Department, Family and Community Medicine Centre, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
| | - Afnan Al-Muhanna
- Breast Imaging Division, Department of Radiology, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
- Radiology Department, King Fahad hospital of the University, Al-Khobar 31952, Saudi Arabia
| | - Arun Vijay Subbarayalu
- Quality Studies and Research Unit, Vice Deanship of Quality, Deanship of Quality and Academic Accreditation, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
| | - Dhoha Al Muhanna
- NDirectorate of Quality and Patient Safety, Family and Community Medicine Center, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
| | - Fahad A. Al-Muhanna
- Nephrology Division, Department of Internal Medicine, Faculty of Medicine, Imam Abdulrahman Bin Faisal University, Dammam 31441, Saudi Arabia
- Medicine Department, King Fahad hospital of the University, Al-Khobar 31952, Saudi Arabia
| |
Collapse
|
22
|
van der Vegt AH, Scott IA, Dermawan K, Schnetler RJ, Kalke VR, Lane PJ. Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework. J Am Med Inform Assoc 2023:7161075. [PMID: 37172264 DOI: 10.1093/jamia/ocad075] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 04/04/2023] [Accepted: 04/23/2023] [Indexed: 05/14/2023] Open
Abstract
OBJECTIVE To retrieve and appraise studies of deployed artificial intelligence (AI)-based sepsis prediction algorithms using systematic methods, identify implementation barriers, enablers, and key decisions and then map these to a novel end-to-end clinical AI implementation framework. MATERIALS AND METHODS Systematically review studies of clinically applied AI-based sepsis prediction algorithms in regard to methodological quality, deployment and evaluation methods, and outcomes. Identify contextual factors that influence implementation and map these factors to the SALIENT implementation framework. RESULTS The review identified 30 articles of algorithms applied in adult hospital settings, with 5 studies reporting significantly decreased mortality post-implementation. Eight groups of algorithms were identified, each sharing a common algorithm. We identified 14 barriers, 26 enablers, and 22 decision points which were able to be mapped to the 5 stages of the SALIENT implementation framework. DISCUSSION Empirical studies of deployed sepsis prediction algorithms demonstrate their potential for improving care and reducing mortality but reveal persisting gaps in existing implementation guidance. In the examined publications, key decision points reflecting real-word implementation experience could be mapped to the SALIENT framework and, as these decision points appear to be AI-task agnostic, this framework may also be applicable to non-sepsis algorithms. The mapping clarified where and when barriers, enablers, and key decisions arise within the end-to-end AI implementation process. CONCLUSIONS A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework that has the ability to account for key factors that warrant attention in ensuring successful deployment, and which extends on previous AI implementation frameworks.
Collapse
Affiliation(s)
- Anton H van der Vegt
- Queensland Digital Health Centre, The University of Queensland, Brisbane, Queensland, Australia
| | - Ian A Scott
- Department of Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Brisbane, Australia
| | - Krishna Dermawan
- Centre for Information Resilience, The University of Queensland, St Lucia, Australia
| | - Rudolf J Schnetler
- School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia, Australia
| | - Vikrant R Kalke
- Patient Safety and Quality, Clinical Excellence Queensland, Queensland Health, Brisbane, Australia
| | - Paul J Lane
- Safety Quality & Innovation, The Prince Charles Hospital, Queensland Health, Brisbane, Australia
| |
Collapse
|
23
|
Masoumian Hosseini M, Masoumian Hosseini ST, Qayumi K, Ahmady S, Koohestani HR. The Aspects of Running Artificial Intelligence in Emergency Care; a Scoping Review. ARCHIVES OF ACADEMIC EMERGENCY MEDICINE 2023; 11:e38. [PMID: 37215232 PMCID: PMC10197918 DOI: 10.22037/aaem.v11i1.1974] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
Abstract
Introduction Artificial Inteligence (AI) application in emergency medicine is subject to ethical and legal inconsistencies. The purposes of this study were to map the extent of AI applications in emergency medicine, to identify ethical issues related to the use of AI, and to propose an ethical framework for its use. Methods A comprehensive literature collection was compiled through electronic databases/internet search engines (PubMed, Web of Science Platform, MEDLINE, Scopus, Google Scholar/Academia, and ERIC) and reference lists. We considered studies published between 1 January 2014 and 6 October 2022. Articles that did not self-classify as studies of an AI intervention, those that were not relevant to Emergency Departments (EDs), and articles that did not report outcomes or evaluations were excluded. Descriptive and thematic analyses of data extracted from the included articles were conducted. Results A total of 137 out of the 2175 citations in the original database were eligible for full-text evaluation. Of these articles, 47 were included in the scoping review and considered for theme extraction. This review covers seven main areas of AI techniques in emergency medicine: Machine Learning (ML) Algorithms (10.64%), prehospital emergency management (12.76%), triage, patient acuity and disposition of patients (19.15%), disease and condition prediction (23.40%), emergency department management (17.03%), the future impact of AI on Emergency Medical Services (EMS) (8.51%), and ethical issues (8.51%). Conclusion There has been a rapid increase in AI research in emergency medicine in recent years. Several studies have demonstrated the potential of AI in diverse contexts, particularly when improving patient outcomes through predictive modelling. According to the synthesis of studies in our review, AI-based decision-making lacks transparency. This feature makes AI decision-making opaque.
Collapse
Affiliation(s)
| | | | - Karim Qayumi
- Centre of Excellence for Simulation Education and Innovation, Department of Surgery, University of British Columbia, Vancouver, BC, Canada
| | - Soleiman Ahmady
- Department of Medical Education, Virtual School of Medical Education & Management, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Hamid Reza Koohestani
- Department of Nursing, Social Determinants of Health Research Center, Saveh University of Medical Sciences, Saveh, Iran
| |
Collapse
|
24
|
Gill SK, Karwath A, Uh HW, Cardoso VR, Gu Z, Barsky A, Slater L, Acharjee A, Duan J, Dall'Olio L, el Bouhaddani S, Chernbumroong S, Stanbury M, Haynes S, Asselbergs FW, Grobbee DE, Eijkemans MJC, Gkoutos GV, Kotecha D. Artificial intelligence to enhance clinical value across the spectrum of cardiovascular healthcare. Eur Heart J 2023; 44:713-725. [PMID: 36629285 PMCID: PMC9976986 DOI: 10.1093/eurheartj/ehac758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/16/2022] [Revised: 11/22/2022] [Accepted: 12/05/2022] [Indexed: 01/12/2023] Open
Abstract
Artificial intelligence (AI) is increasingly being utilized in healthcare. This article provides clinicians and researchers with a step-wise foundation for high-value AI that can be applied to a variety of different data modalities. The aim is to improve the transparency and application of AI methods, with the potential to benefit patients in routine cardiovascular care. Following a clear research hypothesis, an AI-based workflow begins with data selection and pre-processing prior to analysis, with the type of data (structured, semi-structured, or unstructured) determining what type of pre-processing steps and machine-learning algorithms are required. Algorithmic and data validation should be performed to ensure the robustness of the chosen methodology, followed by an objective evaluation of performance. Seven case studies are provided to highlight the wide variety of data modalities and clinical questions that can benefit from modern AI techniques, with a focus on applying them to cardiovascular disease management. Despite the growing use of AI, further education for healthcare workers, researchers, and the public are needed to aid understanding of how AI works and to close the existing gap in knowledge. In addition, issues regarding data access, sharing, and security must be addressed to ensure full engagement by patients and the public. The application of AI within healthcare provides an opportunity for clinicians to deliver a more personalized approach to medical care by accounting for confounders, interactions, and the rising prevalence of multi-morbidity.
Collapse
Affiliation(s)
- Simrat K Gill
- Institute of Cardiovascular Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Andreas Karwath
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Institute of Cancer and Genomic Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
| | - Hae-Won Uh
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Victor Roth Cardoso
- Institute of Cardiovascular Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Institute of Cancer and Genomic Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
| | - Zhujie Gu
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Andrey Barsky
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Institute of Cancer and Genomic Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
| | - Luke Slater
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Institute of Cancer and Genomic Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
| | - Animesh Acharjee
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Institute of Cancer and Genomic Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
| | - Jinming Duan
- School of Computer Science, University of Birmingham, Birmingham, UK
- Alan Turing Institute, London, UK
| | - Lorenzo Dall'Olio
- Department of Physics and Astronomy, University of Bologna, Bologna, Italy
| | - Said el Bouhaddani
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Saisakul Chernbumroong
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Institute of Cancer and Genomic Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
| | | | | | - Folkert W Asselbergs
- Amsterdam University Medical Center, Department of Cardiology, University of Amsterdam, Amsterdam, The Netherlands
- Health Data Research UK and Institute of Health Informatics, University College London, London, UK
| | - Diederick E Grobbee
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Marinus J C Eijkemans
- Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Georgios V Gkoutos
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Institute of Cancer and Genomic Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
| | - Dipak Kotecha
- Institute of Cardiovascular Sciences, University of Birmingham, Vincent Drive, B15 2TT Birmingham, UK
- Health Data Research UK Midlands, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Department of Cardiology, Division Heart and Lungs, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
25
|
Akudjedu TN, Torre S, Khine R, Katsifarakis D, Newman D, Malamateniou C. Knowledge, perceptions, and expectations of Artificial intelligence in radiography practice: A global radiography workforce survey. J Med Imaging Radiat Sci 2023; 54:104-116. [PMID: 36535859 DOI: 10.1016/j.jmir.2022.11.016] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 10/19/2022] [Accepted: 11/29/2022] [Indexed: 12/23/2022]
Abstract
BACKGROUND Artificial Intelligence (AI) technologies have already started impacting clinical practice across various settings worldwide, including the radiography profession. This study is aimed at exploring a world-wide view on AI technologies in relation to knowledge, perceptions, and expectations of radiography professionals. METHODS An online survey (hosted on Qualtrics) on key AI concepts was open to radiography professionals worldwide (August 1st to December 31st 2020). The survey sought both quantitative and qualitative data on topical issues relating to knowledge, perceptions, and expectations in relation to AI implementation in radiography practice. Data obtained was analysed using the Statistical Package for Social Sciences (SPSS) (v.26) and the six-phase thematic analysis approach. RESULTS A total of 314 valid responses were obtained with a fair geographical distribution. Of the respondents, 54.1% (157/290) were from North America and were predominantly clinical practicing radiographers (60.5%, 190/314). Our findings broadly relate to different perceived benefits and misgivings/shortcomings of AI implementation in radiography practice. The benefits relate to enhanced workflows and optimised workstreams while the misgivings/shortcomings revolve around de-skilling and impact on patient-centred care due to over-reliance on advanced technology following AI implementation. DISCUSSION Artificial intelligence is a tool but to operate optimally it requires human input and validation. Radiographers working at the interface between technology and the patient are key stakeholders in AI implementation. Lack of training and of transparency of AI tools create a mixed response of radiographers when they discuss their perceived benefits and challenges. It is also possible that their responses are nuanced by different regional and geographical contexts when it comes to AI deployment. Irrespective of geography, there is still a lot to be done about formalised AI training for radiographers worldwide. This is a vital step to ensure safe and effective AI implementation, adoption, and faster integration into clinical practice by healthcare workers including radiographers. CONCLUSION Advancement of AI technologies and implementation should be accompanied by proportional training of end-users in radiography and beyond. There are many benefits of AI-enabled radiography workflows and improvement on efficiencies but equally there will be widespread disruption of traditional roles and patient-centred care, which can be managed by a well-educated and well-informed workforce.
Collapse
Affiliation(s)
- Theophilus N Akudjedu
- Department of Medical Science and Public Health, Faculty of Health and Social Sciences, Institute of Medical Imaging and Visualisation, Bournemouth University, Bournemouth, Dorset, UK.
| | - Sofia Torre
- Department of Radiography, School of Health Sciences, City, University of London, Northampton Square, London, UK
| | - Ricardo Khine
- School of Health and Care Professions, Buckinghamshire New University, UK
| | | | - Donna Newman
- International Society of Radiographers and Radiological Technologists, UK
| | - Christina Malamateniou
- Department of Radiography, School of Health Sciences, City, University of London, Northampton Square, London, UK
| |
Collapse
|
26
|
khan B, Fatima H, Qureshi A, Kumar S, Hanan A, Hussain J, Abdullah S. Drawbacks of Artificial Intelligence and Their Potential Solutions in the Healthcare Sector. BIOMEDICAL MATERIALS & DEVICES (NEW YORK, N.Y.) 2023; 1:1-8. [PMID: 36785697 PMCID: PMC9908503 DOI: 10.1007/s44174-023-00063-2] [Citation(s) in RCA: 37] [Impact Index Per Article: 37.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Accepted: 01/19/2023] [Indexed: 02/10/2023]
Abstract
Artificial intelligence (AI) has the potential to make substantial progress toward the goal of making healthcare more personalized, predictive, preventative, and interactive. We believe AI will continue its present path and ultimately become a mature and effective tool for the healthcare sector. Besides this AI-based systems raise concerns regarding data security and privacy. Because health records are important and vulnerable, hackers often target them during data breaches. The absence of standard guidelines for the moral use of AI and ML in healthcare has only served to worsen the situation. There is debate about how far artificial intelligence (AI) may be utilized ethically in healthcare settings since there are no universal guidelines for its use. Therefore, maintaining the confidentiality of medical records is crucial. This study enlightens the possible drawbacks of AI in the implementation of healthcare sector and their solutions to overcome these situations. Graphical Abstract
Collapse
Affiliation(s)
- Bangul khan
- Hong Kong Centre for Cerebro-Caradiovasular Health Engineering (COCHE), Shatin, Hong Kong
- Riphah International University, Lahore, Pakistan
| | - Hajira Fatima
- Mehran University of Engineering and Technology, Jamshoro, Pakistan
| | | | | | - Abdul Hanan
- Mehran University of Engineering and Technology, Jamshoro, Pakistan
| | | | - Saad Abdullah
- Riphah International University, Lahore, Pakistan
- Mälardalen University, Västerås, Sweden
| |
Collapse
|
27
|
Nilsen P, Reed J, Nair M, Savage C, Macrae C, Barlow J, Svedberg P, Larsson I, Lundgren L, Nygren J. Realizing the potential of artificial intelligence in healthcare: Learning from intervention, innovation, implementation and improvement sciences. FRONTIERS IN HEALTH SERVICES 2022; 2:961475. [PMID: 36925879 PMCID: PMC10012740 DOI: 10.3389/frhs.2022.961475] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/04/2022] [Accepted: 08/22/2022] [Indexed: 06/18/2023]
Abstract
Introduction Artificial intelligence (AI) is widely seen as critical for tackling fundamental challenges faced by health systems. However, research is scant on the factors that influence the implementation and routine use of AI in healthcare, how AI may interact with the context in which it is implemented, and how it can contribute to wider health system goals. We propose that AI development can benefit from knowledge generated in four scientific fields: intervention, innovation, implementation and improvement sciences. Aim The aim of this paper is to briefly describe the four fields and to identify potentially relevant knowledge from these fields that can be utilized for understanding and/or facilitating the use of AI in healthcare. The paper is based on the authors' experience and expertise in intervention, innovation, implementation, and improvement sciences, and a selective literature review. Utilizing knowledge from the four fields The four fields have generated a wealth of often-overlapping knowledge, some of which we propose has considerable relevance for understanding and/or facilitating the use of AI in healthcare. Conclusion Knowledge derived from intervention, innovation, implementation, and improvement sciences provides a head start for research on the use of AI in healthcare, yet the extent to which this knowledge can be repurposed in AI studies cannot be taken for granted. Thus, when taking advantage of insights in the four fields, it is important to also be explorative and use inductive research approaches to generate knowledge that can contribute toward realizing the potential of AI in healthcare.
Collapse
Affiliation(s)
- Per Nilsen
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden
| | - Julie Reed
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Monika Nair
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Carl Savage
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Department of Learning, Informatics, Management and Ethics, Medical Management Centre, Karolinska Institutet, Stockholm, Sweden
| | - Carl Macrae
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Centre for Health Innovation, Leadership and Learning, Nottingham University Business School, Nottingham, United Kingdom
| | - James Barlow
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Centre for Health Economics and Policy Innovation, Imperial College Business School, London, United Kingdom
| | - Petra Svedberg
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Ingrid Larsson
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Lina Lundgren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Jens Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|
28
|
Petersson L, Larsson I, Nygren JM, Nilsen P, Neher M, Reed JE, Tyskbo D, Svedberg P. Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden. BMC Health Serv Res 2022; 22:850. [PMID: 35778736 PMCID: PMC9250210 DOI: 10.1186/s12913-022-08215-8] [Citation(s) in RCA: 26] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Accepted: 06/20/2022] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Artificial intelligence (AI) for healthcare presents potential solutions to some of the challenges faced by health systems around the world. However, it is well established in implementation and innovation research that novel technologies are often resisted by healthcare leaders, which contributes to their slow and variable uptake. Although research on various stakeholders' perspectives on AI implementation has been undertaken, very few studies have investigated leaders' perspectives on the issue of AI implementation in healthcare. It is essential to understand the perspectives of healthcare leaders, because they have a key role in the implementation process of new technologies in healthcare. The aim of this study was to explore challenges perceived by leaders in a regional Swedish healthcare setting concerning the implementation of AI in healthcare. METHODS The study takes an explorative qualitative approach. Individual, semi-structured interviews were conducted from October 2020 to May 2021 with 26 healthcare leaders. The analysis was performed using qualitative content analysis, with an inductive approach. RESULTS The analysis yielded three categories, representing three types of challenge perceived to be linked with the implementation of AI in healthcare: 1) Conditions external to the healthcare system; 2) Capacity for strategic change management; 3) Transformation of healthcare professions and healthcare practice. CONCLUSIONS In conclusion, healthcare leaders highlighted several implementation challenges in relation to AI within and beyond the healthcare system in general and their organisations in particular. The challenges comprised conditions external to the healthcare system, internal capacity for strategic change management, along with transformation of healthcare professions and healthcare practice. The results point to the need to develop implementation strategies across healthcare organisations to address challenges to AI-specific capacity building. Laws and policies are needed to regulate the design and execution of effective AI implementation strategies. There is a need to invest time and resources in implementation processes, with collaboration across healthcare, county councils, and industry partnerships.
Collapse
Affiliation(s)
- Lena Petersson
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden.
| | - Ingrid Larsson
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden
| | - Jens M Nygren
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden
| | - Per Nilsen
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden.,Department of Health, Medicine and Caring Sciences, Division of Public Health, Faculty of Health Sciences, Linköping University, Linköping, Sweden
| | - Margit Neher
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden.,Department of Rehabilitation, School of Health Sciences, Jönköping University, Jönköping, Sweden
| | - Julie E Reed
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden
| | - Daniel Tyskbo
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden
| | - Petra Svedberg
- School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden
| |
Collapse
|
29
|
Sharma M, Savage C, Nair M, Larsson I, Svedberg P, Nygren JM. Artificial Intelligence Applications in Health Care Practice: A Scoping Review (Preprint). J Med Internet Res 2022; 24:e40238. [PMID: 36197712 PMCID: PMC9582911 DOI: 10.2196/40238] [Citation(s) in RCA: 30] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2022] [Revised: 08/19/2022] [Accepted: 08/30/2022] [Indexed: 11/25/2022] Open
Abstract
Background Artificial intelligence (AI) is often heralded as a potential disruptor that will transform the practice of medicine. The amount of data collected and available in health care, coupled with advances in computational power, has contributed to advances in AI and an exponential growth of publications. However, the development of AI applications does not guarantee their adoption into routine practice. There is a risk that despite the resources invested, benefits for patients, staff, and society will not be realized if AI implementation is not better understood. Objective The aim of this study was to explore how the implementation of AI in health care practice has been described and researched in the literature by answering 3 questions: What are the characteristics of research on implementation of AI in practice? What types and applications of AI systems are described? What characteristics of the implementation process for AI systems are discernible? Methods A scoping review was conducted of MEDLINE (PubMed), Scopus, Web of Science, CINAHL, and PsycINFO databases to identify empirical studies of AI implementation in health care since 2011, in addition to snowball sampling of selected reference lists. Using Rayyan software, we screened titles and abstracts and selected full-text articles. Data from the included articles were charted and summarized. Results Of the 9218 records retrieved, 45 (0.49%) articles were included. The articles cover diverse clinical settings and disciplines; most (32/45, 71%) were published recently, were from high-income countries (33/45, 73%), and were intended for care providers (25/45, 56%). AI systems are predominantly intended for clinical care, particularly clinical care pertaining to patient-provider encounters. More than half (24/45, 53%) possess no action autonomy but rather support human decision-making. The focus of most research was on establishing the effectiveness of interventions (16/45, 35%) or related to technical and computational aspects of AI systems (11/45, 24%). Focus on the specifics of implementation processes does not yet seem to be a priority in research, and the use of frameworks to guide implementation is rare. Conclusions Our current empirical knowledge derives from implementations of AI systems with low action autonomy and approaches common to implementations of other types of information systems. To develop a specific and empirically based implementation framework, further research is needed on the more disruptive types of AI systems being implemented in routine care and on aspects unique to AI implementation in health care, such as building trust, addressing transparency issues, developing explainable and interpretable solutions, and addressing ethical concerns around privacy and data protection.
Collapse
Affiliation(s)
- Malvika Sharma
- Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Medical Management Centre, Stockholm, Sweden
| | - Carl Savage
- Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Medical Management Centre, Stockholm, Sweden
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Monika Nair
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Ingrid Larsson
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Petra Svedberg
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Jens M Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|
30
|
Steerling E, Houston R, Gietzen LJ, Ogilvie SJ, de Ruiter HP, Nygren JM. Examining how ethics in relation to health technology is described in the research literature: A Scoping Review (Preprint). Interact J Med Res 2022; 11:e38745. [PMID: 35969434 PMCID: PMC9425162 DOI: 10.2196/38745] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Revised: 06/06/2022] [Accepted: 07/07/2022] [Indexed: 11/13/2022] Open
Affiliation(s)
- Emilie Steerling
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Rebecca Houston
- College of Allied Health and Nursing, Minnesota State University, Mankato, MN, United States
| | - Luke J Gietzen
- College of Allied Health and Nursing, Minnesota State University, Mankato, MN, United States
| | - Sarah J Ogilvie
- College of Allied Health and Nursing, Minnesota State University, Mankato, MN, United States
| | - Hans-Peter de Ruiter
- College of Allied Health and Nursing, Minnesota State University, Mankato, MN, United States
| | - Jens M Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|
31
|
Svedberg P, Reed J, Nilsen P, Barlow J, Macrae C, Nygren J. Towards successful implementation of artificial intelligence in healthcare practice: A research program (Preprint). JMIR Res Protoc 2021; 11:e34920. [PMID: 35262500 PMCID: PMC8943554 DOI: 10.2196/34920] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Revised: 01/14/2022] [Accepted: 01/19/2022] [Indexed: 02/06/2023] Open
Abstract
Background Objective Methods Results Conclusions International Registered Report Identifier (IRRID)
Collapse
Affiliation(s)
- Petra Svedberg
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Julie Reed
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Per Nilsen
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden
| | - James Barlow
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Centre for Health Economics and Policy Innovation, Imperial College Business School, London, United Kingdom
| | - Carl Macrae
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
- Centre for Health Innovation, Leadership and Learning, Nottingham University Business School, Nottingham, United Kingdom
| | - Jens Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|