1
|
Farah L, Borget I, Martelli N, Vallee A. Suitability of the Current Health Technology Assessment of Innovative Artificial Intelligence-Based Medical Devices: Scoping Literature Review. J Med Internet Res 2024; 26:e51514. [PMID: 38739911 DOI: 10.2196/51514] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 12/17/2023] [Accepted: 12/28/2023] [Indexed: 05/16/2024] Open
Abstract
BACKGROUND Artificial intelligence (AI)-based medical devices have garnered attention due to their ability to revolutionize medicine. Their health technology assessment framework is lacking. OBJECTIVE This study aims to analyze the suitability of each health technology assessment (HTA) domain for the assessment of AI-based medical devices. METHODS We conducted a scoping literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We searched databases (PubMed, Embase, and Cochrane Library), gray literature, and HTA agency websites. RESULTS A total of 10.1% (78/775) of the references were included. Data quality and integration are vital aspects to consider when describing and assessing the technical characteristics of AI-based medical devices during an HTA process. When it comes to implementing specialized HTA for AI-based medical devices, several practical challenges and potential barriers could be highlighted and should be taken into account (AI technological evolution timeline, data requirements, complexity and transparency, clinical validation and safety requirements, regulatory and ethical considerations, and economic evaluation). CONCLUSIONS The adaptation of the HTA process through a methodological framework for AI-based medical devices enhances the comparability of results across different evaluations and jurisdictions. By defining the necessary expertise, the framework supports the development of a skilled workforce capable of conducting robust and reliable HTAs of AI-based medical devices. A comprehensive adapted HTA framework for AI-based medical devices can provide valuable insights into the effectiveness, cost-effectiveness, and societal impact of AI-based medical devices, guiding their responsible implementation and maximizing their benefits for patients and health care systems.
Collapse
Affiliation(s)
- Line Farah
- Innovation Center for Medical Devices Department, Foch Hospital, Suresnes, France
- Groupe de Recherche et d'accueil en Droit et Economie de la Santé Department, University Paris-Saclay, Orsay, France
| | - Isabelle Borget
- Groupe de Recherche et d'accueil en Droit et Economie de la Santé Department, University Paris-Saclay, Orsay, France
- Department of Biostatistics and Epidemiology, Gustave Roussy, University Paris-Saclay, Villejuif, France
- Oncostat U1018, Inserm, Équipe Labellisée Ligue Contre le Cancer, University Paris-Saclay, Villejuif, France
| | - Nicolas Martelli
- Groupe de Recherche et d'accueil en Droit et Economie de la Santé Department, University Paris-Saclay, Orsay, France
- Pharmacy Department, Georges Pompidou European Hospital, Paris, France
| | - Alexandre Vallee
- Department of Epidemiology and Public Health, Foch Hospital, Suresnes, France
| |
Collapse
|
2
|
Singh S, Moore E, Melissa P, Patel V, Brown J, Davidson J. Initial evaluation of a technologyenabled change in delivery of the dementia service during COVID-19 in North Warwickshire. Br J Community Nurs 2024; 29:224-230. [PMID: 38701016 DOI: 10.12968/bjcn.2024.29.5.224] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/05/2024]
Abstract
BACKGROUND Remote monitoring technologies show potential to help health professionals deliver preventative interventions which can avoid hospital admissions and allow patients to remain in a home setting. AIMS To assess whether an Internet of Things (IoT) driven remote monitoring technology, used in the care pathway of community dementia patients in North Warwickshire improved access to care for patients and cost effectiveness. METHOD Patient level changes to anonymised retrospective healthcare utilisation data were analysed alongside costs. RESULTS Urgent care decreased following use of an IoT driven remote monitoring technology; one preventative intervention avoided an average of three urgent interventions. A Chi-Square test showing this change as significant. Estimates show annualised service activity avoidance of £201,583 for the cohort; £8764 per patient. CONCLUSIONS IoT driven remote monitoring had a positive impact on health utilisation and cost avoidance. Future expansion of the cohort will allow for validation of the results and consider the impact of the technology on patient health outcomes and staff workflows.
Collapse
Affiliation(s)
- Sid Singh
- Department of Clinical Informatics George Eliot Hospital NHS Trust, Nuneaton UK
| | | | - Paolo Melissa
- Department of Clinical Informatics George Eliot Hospital NHS Trust, Nuneaton UK
| | - Vinod Patel
- Department of Clinical Informatics George Eliot Hospital NHS Trust, Nuneaton UK
| | | | - Jan Davidson
- Warwick Manufacturing Group, Warwick University UK
| |
Collapse
|
3
|
Zwerwer LR, van der Pol S, Zacharowski K, Postma MJ, Kloka J, Friedrichson B, van Asselt ADI. The value of artificial intelligence for the treatment of mechanically ventilated intensive care unit patients: An early health technology assessment. J Crit Care 2024; 82:154802. [PMID: 38583302 DOI: 10.1016/j.jcrc.2024.154802] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Revised: 03/03/2024] [Accepted: 03/23/2024] [Indexed: 04/09/2024]
Abstract
PURPOSE The health and economic consequences of artificial intelligence (AI) systems for mechanically ventilated intensive care unit patients often remain unstudied. Early health technology assessments (HTA) can examine the potential impact of AI systems by using available data and simulations. Therefore, we developed a generic health-economic model suitable for early HTA of AI systems for mechanically ventilated patients. MATERIALS AND METHODS Our generic health-economic model simulates mechanically ventilated patients from their hospitalisation until their death. The model simulates two scenarios, care as usual and care with the AI system, and compares these scenarios to estimate their cost-effectiveness. RESULTS The generic health-economic model we developed is suitable for estimating the cost-effectiveness of various AI systems. By varying input parameters and assumptions, the model can examine the cost-effectiveness of AI systems across a wide range of different clinical settings. CONCLUSIONS Using the proposed generic health-economic model, investors and innovators can easily assess whether implementing a certain AI system is likely to be cost-effective before an exact clinical impact is determined. The results of the early HTA can aid investors and innovators in deployment of AI systems by supporting development decisions, informing value-based pricing, clinical trial design, and selection of target patient groups.
Collapse
Affiliation(s)
- Leslie R Zwerwer
- Department of Health Sciences, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands.
| | - Simon van der Pol
- Department of Health Sciences, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands; Health-Ecore, Zeist, the Netherlands
| | - Kai Zacharowski
- Department of Anaesthesiology, Intensive Care Medicine and Pain Therapy, University Hospital Frankfurt, Goethe University, Frankfurt, Germany
| | - Maarten J Postma
- Department of Health Sciences, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands; Health-Ecore, Zeist, the Netherlands; Department of Economics, Econometrics and Finance, University of Groningen, Faculty of Economics and Business, Groningen, the Netherlands; Center of Excellence for Pharmaceutical Care, Universitas Padjadjaran, Bandung, Indonesia
| | - Jan Kloka
- Department of Anaesthesiology, Intensive Care Medicine and Pain Therapy, University Hospital Frankfurt, Goethe University, Frankfurt, Germany
| | - Benjamin Friedrichson
- Department of Anaesthesiology, Intensive Care Medicine and Pain Therapy, University Hospital Frankfurt, Goethe University, Frankfurt, Germany
| | - Antoinette D I van Asselt
- Department of Health Sciences, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands; Department of Epidemiology, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands
| |
Collapse
|
4
|
Natesan D, Eisenstein EL, Thomas SM, Eclov NCW, Dalal NH, Stephens SJ, Malicki M, Shields S, Cobb A, Mowery YM, Niedzwiecki D, Tenenbaum JD, Palta M, Hong JC. Health Care Cost Reductions with Machine Learning-Directed Evaluations during Radiation Therapy - An Economic Analysis of a Randomized Controlled Study. NEJM AI 2024; 1:10.1056/aioa2300118. [PMID: 38586278 PMCID: PMC10997376 DOI: 10.1056/aioa2300118] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
BACKGROUND Machine learning (ML) may cost-effectively direct health care by identifying patients most likely to benefit from preventative interventions to avoid negative and expensive outcomes. System for High-Intensity Evaluation During Radiation Therapy (SHIELD-RT; NCT04277650) was a single-institution, randomized controlled study in which electronic health record-based ML accurately identified patients at high risk for acute care (emergency visit or hospitalization) during radiotherapy (RT) and targeted them for supplemental clinical evaluations. This ML-directed intervention resulted in decreased acute care utilization. Given the limited prospective data showing the ability of ML to direct interventions cost-efficiently, an economic analysis was performed. METHODS A post hoc economic analysis was conducted of SHIELD-RT that included RT courses from January 7, 2019, to June 30, 2019. ML-identified high-risk courses (≥10% risk of acute care during RT) were randomized to receive standard of care weekly clinical evaluations with ad hoc supplemental evaluations per clinician discretion versus mandatory twice-weekly evaluations. The primary outcome was difference in mean total medical costs during and 15 days after RT. Acute care costs were obtained via institutional cost accounting. Physician and intervention costs were estimated via Medicare and Medicaid data. Negative binomial regression was used to estimate cost outcomes after adjustment for patient and disease factors. RESULTS A total of 311 high-risk RT courses among 305 patients were randomized to the standard (n=157) or the intervention (n=154) group. Unadjusted mean intervention group supplemental visit costs were $155 per course (95% confidence interval, $142 to $168). The intervention group had fewer acute care visits per course (standard, 0.47; intervention, 0.31; P=0.04). Total mean adjusted costs were $3110 per course for the standard group and $1494 for the intervention group (difference in means, $1616 [95% confidence interval, $1450 to $1783]; P=0.03). CONCLUSIONS In this economic analysis of a randomized controlled, health care ML study, mandatory supplemental evaluations for ML-identified high-risk patients were associated with both reduced total medical costs and improved clinical outcomes. Further study is needed to determine whether economic results are generalizable. (Funded in part by The Duke Endowment, The Conquer Cancer Foundation, the Duke Department of Radiation Oncology, and the National Cancer Institute of the National Institutes of Health [R01CA277782]; ClinicalTrials.gov number, NCT04277650.).
Collapse
Affiliation(s)
- Divya Natesan
- Department of Radiation Oncology, University of North Carolina, Chapel Hill, NC
- Department of Radiation Oncology, Duke University, Durham, NC
| | | | - Samantha M Thomas
- Department of Biostatistics and Bioinformatics, Duke University, Durham, NC
- Duke Cancer Institute, Duke University, Durham, NC
| | | | - Nicole H Dalal
- Department of Radiation Oncology, Duke University, Durham, NC
| | | | - Mary Malicki
- Department of Radiation Oncology, Duke University, Durham, NC
| | - Stacey Shields
- Department of Radiation Oncology, Duke University, Durham, NC
| | - Alyssa Cobb
- Department of Radiation Oncology, Duke University, Durham, NC
| | - Yvonne M Mowery
- Department of Radiation Oncology, Duke University, Durham, NC
- Duke Cancer Institute, Duke University, Durham, NC
| | - Donna Niedzwiecki
- Department of Biostatistics and Bioinformatics, Duke University, Durham, NC
- Duke Cancer Institute, Duke University, Durham, NC
| | | | - Manisha Palta
- Department of Radiation Oncology, Duke University, Durham, NC
- Duke Cancer Institute, Duke University, Durham, NC
| | - Julian C Hong
- Department of Radiation Oncology, University of California, San Francisco, San Francisco
- Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco
- UCSF-UC Berkeley Joint Program in Computational Precision Health, San Francisco, San Francisco
| |
Collapse
|
5
|
Bharadwaj P, Nicola L, Breau-Brunel M, Sensini F, Tanova-Yotova N, Atanasov P, Lobig F, Blankenburg M. Unlocking the Value: Quantifying the Return on Investment of Hospital Artificial Intelligence. J Am Coll Radiol 2024:S1546-1440(24)00292-8. [PMID: 38499053 DOI: 10.1016/j.jacr.2024.02.034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Revised: 02/23/2024] [Accepted: 02/28/2024] [Indexed: 03/20/2024]
Abstract
PURPOSE A comprehensive return on investment (ROI) calculator was developed to evaluate the monetary and nonmonetary benefits of an artificial intelligence (AI)-powered radiology diagnostic imaging platform to inform decision makers interested in adopting AI. METHODS A calculator was constructed to calculate comparative costs, estimated revenues, and quantify the clinical value of using an AI platform compared with no use of AI in radiology workflows of a US hospital over a 5-year time horizon. Parameters were determined on the basis of expert interviews and a literature review. Scenario and deterministic sensitivity analyses were conducted to evaluate calculator drivers. RESULTS In the calculator, the introduction of an AI platform into the hospital radiology workflow resulted in labor time reductions and delivery of an ROI of 451% over a 5-year period. The ROI was increased to 791% when radiologist time savings were considered. Time savings for radiologists included more than 15 8-hour working days of waiting time, 78 days in triage time, 10 days in reading time, and 41 days in reporting time. Using the platform also provided revenue benefits for the hospital in bringing in patients for clinically beneficial follow-up scans, hospitalizations, and treatment procedures. Results were sensitive to the time horizon, health center setting, and number of scans performed. Among those, the most influential outcome was the number of additional necessary treatments performed because of AI identification of patients. CONCLUSIONS The authors demonstrate a substantial 5-year ROI of implementing an AI platform in a stroke management-accredited hospital. The ROI calculator may be useful for decision makers evaluating AI-powered radiology platforms.
Collapse
Affiliation(s)
| | - Lauren Nicola
- CEO/Partner, Triad Radiology Associates; Chair, Ultrasound Commission, ACR; Chair, Reimbursement Committee, ACR
| | | | | | | | - Petar Atanasov
- Principal Consultant, Amaris Consulting, London, United Kingdom
| | | | | |
Collapse
|
6
|
Cruz A, Dias EM, Scoton MLD, Branco BHM. Health 4.0 in the medical sector: a narrative review. REVISTA DA ASSOCIACAO MEDICA BRASILEIRA (1992) 2024; 70:e20231149. [PMID: 38511760 PMCID: PMC10941873 DOI: 10.1590/1806-9282.20231149] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/27/2023] [Accepted: 09/24/2023] [Indexed: 03/22/2024]
Affiliation(s)
- Antônio Cruz
- Universidade de Sao Paulo, Institute of Radiology, Clinical Hospital, Faculty of Medicine – São Paulo (SP), Brazil
| | - Eduardo Mário Dias
- Universidade de Sao Paulo, Polytechnic School, Department of Electrical Energy Engineering and Automation – São Paulo (SP), Brazil
| | - Maria Lídia Dias Scoton
- Universidade de Sao Paulo, Electrical Automation in Industrial Systems Group – São Paulo (SP), Brazil
| | | |
Collapse
|
7
|
Boverhof BJ, Redekop WK, Bos D, Starmans MPA, Birch J, Rockall A, Visser JJ. Radiology AI Deployment and Assessment Rubric (RADAR) to bring value-based AI into radiological practice. Insights Imaging 2024; 15:34. [PMID: 38315288 PMCID: PMC10844175 DOI: 10.1186/s13244-023-01599-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Accepted: 11/14/2023] [Indexed: 02/07/2024] Open
Abstract
OBJECTIVE To provide a comprehensive framework for value assessment of artificial intelligence (AI) in radiology. METHODS This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury's imaging efficacy framework to facilitate the valuation of radiology AI from conception to local implementation. Local efficacy has been newly introduced to underscore the importance of appraising an AI technology within its local environment. Furthermore, the RADAR framework is illustrated through a myriad of study designs that help assess value. RESULTS RADAR presents a seven-level hierarchy, providing radiologists, researchers, and policymakers with a structured approach to the comprehensive assessment of value in radiology AI. RADAR is designed to be dynamic and meet the different valuation needs throughout the AI's lifecycle. Initial phases like technical and diagnostic efficacy (RADAR-1 and RADAR-2) are assessed pre-clinical deployment via in silico clinical trials and cross-sectional studies. Subsequent stages, spanning from diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5), require clinical integration and are explored via randomized controlled trials and cohort studies. Cost-effectiveness efficacy (RADAR-6) takes a societal perspective on financial feasibility, addressed via health-economic evaluations. The final level, RADAR-7, determines how prior valuations translate locally, evaluated through budget impact analysis, multi-criteria decision analyses, and prospective monitoring. CONCLUSION The RADAR framework offers a comprehensive framework for valuing radiology AI. Its layered, hierarchical structure, combined with a focus on local relevance, aligns RADAR seamlessly with the principles of value-based radiology. CRITICAL RELEVANCE STATEMENT The RADAR framework advances artificial intelligence in radiology by delineating a much-needed framework for comprehensive valuation. KEYPOINTS • Radiology artificial intelligence lacks a comprehensive approach to value assessment. • The RADAR framework provides a dynamic, hierarchical method for thorough valuation of radiology AI. • RADAR advances clinical radiology by bridging the artificial intelligence implementation gap.
Collapse
Affiliation(s)
- Bart-Jan Boverhof
- Erasmus School of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - W Ken Redekop
- Erasmus School of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Daniel Bos
- Department of Epidemiology, Erasmus University Medical Centre, Rotterdam, The Netherlands
- Department of Radiology & Nuclear Medicine, Erasmus University Medical Centre, Rotterdam, The Netherlands
| | - Martijn P A Starmans
- Department of Radiology & Nuclear Medicine, Erasmus University Medical Centre, Rotterdam, The Netherlands
| | | | - Andrea Rockall
- Department of Surgery & Cancer, Imperial College London, London, UK
| | - Jacob J Visser
- Department of Radiology & Nuclear Medicine, Erasmus University Medical Centre, Rotterdam, The Netherlands.
| |
Collapse
|
8
|
Ruamviboonsuk P, Ruamviboonsuk V, Tiwari R. Recent evidence of economic evaluation of artificial intelligence in ophthalmology. Curr Opin Ophthalmol 2023; 34:449-458. [PMID: 37459289 DOI: 10.1097/icu.0000000000000987] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/12/2023]
Abstract
PURPOSE OF REVIEW Health economic evaluation (HEE) is essential for assessing value of health interventions, including artificial intelligence. Recent approaches, current challenges, and future directions of HEE of artificial intelligence in ophthalmology are reviewed. RECENT FINDINGS Majority of recent HEEs of artificial intelligence in ophthalmology were for diabetic retinopathy screening. Two models, one conducted in the rural USA (5-year period) and another in China (35-year period), found artificial intelligence to be more cost-effective than without screening for diabetic retinopathy. Two additional models, which compared artificial intelligence with human screeners in Brazil and Thailand for the lifetime of patients, found artificial intelligence to be more expensive from a healthcare system perspective. In the Thailand analysis, however, artificial intelligence was less expensive when opportunity loss from blindness was included. An artificial intelligence model for screening retinopathy of prematurity was cost-effective in the USA. A model for screening age-related macular degeneration in Japan and another for primary angle close in China did not find artificial intelligence to be cost-effective, compared with no screening. The costs of artificial intelligence varied widely in these models. SUMMARY Like other medical fields, there is limited evidence in assessing the value of artificial intelligence in ophthalmology and more appropriate HEE models are needed.
Collapse
Affiliation(s)
- Paisan Ruamviboonsuk
- Department of Ophthalmology, Rajavithi Hospital, College of Medicine, Rangsit University
| | | | | |
Collapse
|
9
|
Vithlani J, Hawksworth C, Elvidge J, Ayiku L, Dawoud D. Economic evaluations of artificial intelligence-based healthcare interventions: a systematic literature review of best practices in their conduct and reporting. Front Pharmacol 2023; 14:1220950. [PMID: 37693892 PMCID: PMC10486896 DOI: 10.3389/fphar.2023.1220950] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Accepted: 07/25/2023] [Indexed: 09/12/2023] Open
Abstract
Objectives: Health economic evaluations (HEEs) help healthcare decision makers understand the value of new technologies. Artificial intelligence (AI) is increasingly being used in healthcare interventions. We sought to review the conduct and reporting of published HEEs for AI-based health interventions. Methods: We conducted a systematic literature review with a 15-month search window (April 2021 to June 2022) on 17th June 2022 to identify HEEs of AI health interventions and update a previous review. Records were identified from 3 databases (Medline, Embase, and Cochrane Central). Two reviewers screened papers against predefined study selection criteria. Data were extracted from included studies using prespecified data extraction tables. Included studies were quality assessed using the National Institute for Health and Care Excellence (NICE) checklist. Results were synthesized narratively. Results: A total of 21 studies were included. The most common type of AI intervention was automated image analysis (9/21, 43%) mainly used for screening or diagnosis in general medicine and oncology. Nearly all were cost-utility (10/21, 48%) or cost-effectiveness analyses (8/21, 38%) that took a healthcare system or payer perspective. Decision-analytic models were used in 16/21 (76%) studies, mostly Markov models and decision trees. Three (3/16, 19%) used a short-term decision tree followed by a longer-term Markov component. Thirteen studies (13/21, 62%) reported the AI intervention to be cost effective or dominant. Limitations tended to result from the input data, authorship conflicts of interest, and a lack of transparent reporting, especially regarding the AI nature of the intervention. Conclusion: Published HEEs of AI-based health interventions are rapidly increasing in number. Despite the potentially innovative nature of AI, most have used traditional methods like Markov models or decision trees. Most attempted to assess the impact on quality of life to present the cost per QALY gained. However, studies have not been comprehensively reported. Specific reporting standards for the economic evaluation of AI interventions would help improve transparency and promote their usefulness for decision making. This is fundamental for reimbursement decisions, which in turn will generate the necessary data to develop flexible models better suited to capturing the potentially dynamic nature of AI interventions.
Collapse
Affiliation(s)
- Jai Vithlani
- National Institute for Health and Care Excellence, London, United Kingdom
| | - Claire Hawksworth
- National Institute for Health and Care Excellence, Manchester, United Kingdom
| | - Jamie Elvidge
- National Institute for Health and Care Excellence, Manchester, United Kingdom
| | - Lynda Ayiku
- National Institute for Health and Care Excellence, Manchester, United Kingdom
| | - Dalia Dawoud
- National Institute for Health and Care Excellence, London, United Kingdom
- Faculty of Pharmacy, Cairo University, Cairo, Egypt
| |
Collapse
|
10
|
van Leeuwen K, Becks M, Grob D, de Lange F, Rutten J, Schalekamp S, Rutten M, van Ginneken B, de Rooij M, Meijer F. AI-support for the detection of intracranial large vessel occlusions: One-year prospective evaluation. Heliyon 2023; 9:e19065. [PMID: 37636476 PMCID: PMC10458691 DOI: 10.1016/j.heliyon.2023.e19065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Revised: 08/07/2023] [Accepted: 08/09/2023] [Indexed: 08/29/2023] Open
Abstract
Purpose Few studies have evaluated real-world performance of radiological AI-tools in clinical practice. Over one-year, we prospectively evaluated the use of AI software to support the detection of intracranial large vessel occlusions (LVO) on CT angiography (CTA). Method Quantitative measures (user log-in attempts, AI standalone performance) and qualitative data (user surveys) were reviewed by a key-user group at three timepoints. A total of 491 CTA studies of 460 patients were included for analysis. Results The overall accuracy of the AI-tool for LVO detection and localization was 87.6%, sensitivity 69.1% and specificity 91.2%. Out of 81 LVOs, 31 of 34 (91%) M1 occlusions were detected correctly, 19 of 38 (50%) M2 occlusions, and 6 of 9 (67%) ICA occlusions. The product was considered user-friendly. The diagnostic confidence of the users for LVO detection remained the same over the year. The last measured net promotor score was -56%. The use of the AI-tool fluctuated over the year with a declining trend. Conclusions Our pragmatic approach of evaluating the AI-tool used in clinical practice, helped us to monitor the usage, to estimate the perceived added value by the users of the AI-tool, and to make an informed decision about the continuation of the use of the AI-tool.
Collapse
Affiliation(s)
- K.G. van Leeuwen
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - M.J. Becks
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - D. Grob
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - F. de Lange
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - J.H.E. Rutten
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - S. Schalekamp
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - M.J.C.M. Rutten
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
- Department of Radiology, Jeroen Bosch Hospital, ‘s-Hertogenbosch, the Netherlands
| | - B. van Ginneken
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - M. de Rooij
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| | - F.J.A. Meijer
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, the Netherlands
| |
Collapse
|
11
|
Oberije CJG, Sharma N, James JJ, Ng AY, Nash J, Kecskemethy PD. Comparing Prognostic Factors of Cancers Identified by Artificial Intelligence (AI) and Human Readers in Breast Cancer Screening. Cancers (Basel) 2023; 15:3069. [PMID: 37370680 DOI: 10.3390/cancers15123069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Revised: 05/25/2023] [Accepted: 06/02/2023] [Indexed: 06/29/2023] Open
Abstract
Invasiveness status, histological grade, lymph node stage, and tumour size are important prognostic factors for breast cancer survival. This evaluation aims to compare these features for cancers detected by AI and human readers using digital mammography. Women diagnosed with breast cancer between 2009 and 2019 from three UK double-reading sites were included in this retrospective cohort evaluation. Differences in prognostic features of cancers detected by AI and the first human reader (R1) were assessed using chi-square tests, with significance at p < 0.05. From 1718 screen-detected cancers (SDCs) and 293 interval cancers (ICs), AI flagged 85.9% and 31.7%, respectively. R1 detected 90.8% of SDCs and 7.2% of ICs. Of the screen-detected cancers detected by the AI, 82.5% had an invasive component, compared to 81.1% for R1 (p-0.374). For the ICs, this was 91.5% and 93.8% for AI and R1, respectively (p = 0.829). For the invasive tumours, no differences were found for histological grade, tumour size, or lymph node stage. The AI detected more ICs. In summary, no differences in prognostic factors were found comparing SDC and ICs identified by AI or human readers. These findings support a potential role for AI in the double-reading workflow.
Collapse
Affiliation(s)
- Cary J G Oberije
- Kheiron Medical Technologies, 112-116 Old St., London EC1V 9BG, UK
| | - Nisha Sharma
- Breast Screening Unit, Leeds Teaching Hospital NHS Trust, Leeds LS14 6UH, UK
| | - Jonathan J James
- Nottingham Breast Institute, City Hospital, Nottingham University Hospitals NHS Trust, Nottingham NG5 1PB, UK
| | - Annie Y Ng
- Kheiron Medical Technologies, 112-116 Old St., London EC1V 9BG, UK
| | - Jonathan Nash
- Kheiron Medical Technologies, 112-116 Old St., London EC1V 9BG, UK
| | | |
Collapse
|
12
|
Wawer Matos PA, Reimer RP, Rokohl AC, Caldeira L, Heindl LM, Große Hokamp N. Artificial Intelligence in Ophthalmology - Status Quo and Future Perspectives. Semin Ophthalmol 2023; 38:226-237. [PMID: 36356300 DOI: 10.1080/08820538.2022.2139625] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
Artificial intelligence (AI) is an emerging technology in healthcare and holds the potential to disrupt many arms in medical care. In particular, disciplines using medical imaging modalities, including e.g. radiology but ophthalmology as well, are already confronted with a wide variety of AI implications. In ophthalmologic research, AI has demonstrated promising results limited to specific diseases and imaging tools, respectively. Yet, implementation of AI in clinical routine is not widely spread due to availability, heterogeneity in imaging techniques and AI methods. In order to describe the status quo, this narrational review provides a brief introduction to AI ("what the ophthalmologist needs to know"), followed by an overview of different AI-based applications in ophthalmology and a discussion on future challenges.Abbreviations: Age-related macular degeneration, AMD; Artificial intelligence, AI; Anterior segment OCT, AS-OCT; Coronary artery calcium score, CACS; Convolutional neural network, CNN; Deep convolutional neural network, DCNN; Diabetic retinopathy, DR; Machine learning, ML; Optical coherence tomography, OCT; Retinopathy of prematurity, ROP; Support vector machine, SVM; Thyroid-associated ophthalmopathy, TAO.
Collapse
Affiliation(s)
| | - Robert P Reimer
- Department of Diagnostic and Interventional Radiology, University Hospital of Cologne, Köln, Germany
| | - Alexander C Rokohl
- Department of Ophthalmology, University Hospital of Cologne, Köln, Germany
| | - Liliana Caldeira
- Department of Diagnostic and Interventional Radiology, University Hospital of Cologne, Köln, Germany
| | - Ludwig M Heindl
- Department of Ophthalmology, University Hospital of Cologne, Köln, Germany
| | - Nils Große Hokamp
- Department of Diagnostic and Interventional Radiology, University Hospital of Cologne, Köln, Germany
| |
Collapse
|
13
|
Mori Y, East JE, Hassan C, Halvorsen N, Berzin TM, Byrne M, von Renteln D, Hewett DG, Repici A, Ramchandani M, Al Khatry M, Kudo SE, Wang P, Yu H, Saito Y, Misawa M, Parasa S, Matsubayashi CO, Ogata H, Tajiri H, Pausawasdi N, Dekker E, Ahmad OF, Sharma P, Rex DK. Benefits and challenges in implementation of artificial intelligence in colonoscopy: World Endoscopy Organization position statement. Dig Endosc 2023; 35:422-429. [PMID: 36749036 DOI: 10.1111/den.14531] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Accepted: 02/06/2023] [Indexed: 02/08/2023]
Abstract
The number of artificial intelligence (AI) tools for colonoscopy on the market is increasing with supporting clinical evidence. Nevertheless, their implementation is not going smoothly for a variety of reasons, including lack of data on clinical benefits and cost-effectiveness, lack of trustworthy guidelines, uncertain indications, and cost for implementation. To address this issue and better guide practitioners, the World Endoscopy Organization (WEO) has provided its perspective about the status of AI in colonoscopy as the position statement. WEO Position Statement: Statement 1.1: Computer-aided detection (CADe) for colorectal polyps is likely to improve colonoscopy effectiveness by reducing adenoma miss rates and thus increase adenoma detection; Statement 1.2: In the short term, use of CADe is likely to increase health-care costs by detecting more adenomas; Statement 1.3: In the long term, the increased cost by CADe could be balanced by savings in costs related to cancer treatment (surgery, chemotherapy, palliative care) due to CADe-related cancer prevention; Statement 1.4: Health-care delivery systems and authorities should evaluate the cost-effectiveness of CADe to support its use in clinical practice; Statement 2.1: Computer-aided diagnosis (CADx) for diminutive polyps (≤5 mm), when it has sufficient accuracy, is expected to reduce health-care costs by reducing polypectomies, pathological examinations, or both; Statement 2.2: Health-care delivery systems and authorities should evaluate the cost-effectiveness of CADx to support its use in clinical practice; Statement 3: We recommend that a broad range of high-quality cost-effectiveness research should be undertaken to understand whether AI implementation benefits populations and societies in different health-care systems.
Collapse
Affiliation(s)
- Yuichi Mori
- Clinical Effectiveness Research Group, University of Oslo, Oslo, Norway.,Department of Transplantation Medicine, Oslo University Hospital, Oslo, Norway.,Digestive Disease Center, Showa University Northern Yokohama Hospital, Kanagawa, Japan
| | - James E East
- Translational Gastroenterology Unit, John Radcliffe Hospital, University of Oxford, Oxford, UK.,NIHR Oxford Biomedical Research Centre, Oxford, UK.,Division of Gastroenterology and Hepatology, Mayo Clinic Healthcare, London, UK
| | - Cesare Hassan
- Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Italy.,Endoscopy Unit, Humanitas Clinical and Research Center - IRCCS, Rozzano, Italy
| | - Natalie Halvorsen
- Clinical Effectiveness Research Group, University of Oslo, Oslo, Norway
| | - Tyler M Berzin
- Division of Gastroenterology, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, USA
| | - Michael Byrne
- Department of Medicine, The University of British Columbia, Vancouver, Canada
| | - Daniel von Renteln
- Division of Gastroenterology, University of Montreal Medical Center (CHUM) and Research Center (CRCHUM), Montreal, Canada
| | - David G Hewett
- School of Medicine, The University of Queensland, Brisbane, Australia
| | - Alessandro Repici
- Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Italy.,Endoscopy Unit, Humanitas Clinical and Research Center - IRCCS, Rozzano, Italy
| | | | - Maryam Al Khatry
- Department of Gastroenterology, Obaidulla Hospital, Ras Al Khaimah, United Arab Emirates
| | - Shin-Ei Kudo
- Digestive Disease Center, Showa University Northern Yokohama Hospital, Kanagawa, Japan
| | - Pu Wang
- Sichuan Academy of Medical Sciences and Sichuan Provincial People's Hospital, Chengdu, China
| | - Honggang Yu
- Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China
| | - Yutaka Saito
- Endoscopy Division, National Cancer Center Hospital, Tokyo, Japan
| | - Masashi Misawa
- Digestive Disease Center, Showa University Northern Yokohama Hospital, Kanagawa, Japan
| | | | - Carolina Ogawa Matsubayashi
- Gastrointestinal Endoscopy Unit, Gastroenterology Department, University of São Paulo Medical School, São Paulo, Brazil
| | - Haruhiko Ogata
- Center for Diagnostic and Therapeutic Endoscopy, School of Medicine, Keio University, Tokyo, Japan
| | - Hisao Tajiri
- Jikei University School of Medicine, Tokyo, Japan
| | - Nonthalee Pausawasdi
- Vikit Viranuvatti Siriraj GI Endoscopy Center,, Mahidol University, Bangkok, Thailand.,Division of Gastroenterology, Department of Medicine, Faculty of Medicine, Siriraj Hospital, Mahidol University, Bangkok, Thailand
| | - Evelien Dekker
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Center, Amsterdam, The Netherlands
| | | | - Prateek Sharma
- Division of Gastroenterology and Hepatology, University of Kansas School of Medicine and VA Medical Center, Kansas City, USA
| | - Douglas K Rex
- Division of Gastroenterology, Indiana University School of Medicine, Indianapolis, USA
| |
Collapse
|
14
|
Minghui Y, Hu Y, Lu Z. How do nurses work in chronic management in the age of artificial intelligence? development and future prospects. Digit Health 2023; 9:20552076231221057. [PMID: 38116395 PMCID: PMC10729617 DOI: 10.1177/20552076231221057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2022] [Accepted: 11/30/2023] [Indexed: 12/21/2023] Open
Abstract
AI is undeniably revolutionizing medical research and patient care across diverse fields. Chronic disease nursing care, a pivotal aspect of clinical management, has significantly reaped the benefits of AI across numerous dimensions. Understanding the operational principles of artificial intelligence before implementation is crucial, avoiding indiscriminate replacement of all tasks with AI. Nurses serve as the primary force in symptom group research, expanding beyond diabetes to encompass various chronic diseases; their primary responsibility involves recording patients' daily symptoms and vital signs. However, a substantial portion of current AI research excludes nurses from the developmental phase, encompassing them solely in user and feedback populations. The comprehensive design of the symptom analysis and long-term management approach necessitates the guidance and oversight of nurses; however, their current insufficient involvement might stem from nursing staff's comparatively limited comprehension of AI and their ambiguous perception of their role's value in AI. Therefore, an imperative exploration of nurses' roles in symptom analysis and long-term management, leveraging the latest research in these areas, is vital to pinpoint breakthroughs in nurses' AI involvement in the future.
Collapse
Affiliation(s)
- Ye Minghui
- First author: Nursing Administration department, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Yingying Hu
- The First Affiliated Hospital of Wenzhou Medical University, Emergency Department, Wenzhou, Zhejiang, China
| | - Zhongiu Lu
- The First Affiliated Hospital of Wenzhou Medical University, Emergency Department, Wenzhou, Zhejiang, China
| |
Collapse
|
15
|
Zemplényi A, Tachkov K, Balkanyi L, Németh B, Petykó ZI, Petrova G, Czech M, Dawoud D, Goettsch W, Gutierrez Ibarluzea I, Hren R, Knies S, Lorenzovici L, Maravic Z, Piniazhko O, Savova A, Manova M, Tesar T, Zerovnik S, Kaló Z. Recommendations to overcome barriers to the use of artificial intelligence-driven evidence in health technology assessment. Front Public Health 2023; 11:1088121. [PMID: 37181704 PMCID: PMC10171457 DOI: 10.3389/fpubh.2023.1088121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Accepted: 04/03/2023] [Indexed: 05/16/2023] Open
Abstract
Background Artificial intelligence (AI) has attracted much attention because of its enormous potential in healthcare, but uptake has been slow. There are substantial barriers that challenge health technology assessment (HTA) professionals to use AI-generated evidence for decision-making from large real-world databases (e.g., based on claims data). As part of the European Commission-funded HTx H2020 (Next Generation Health Technology Assessment) project, we aimed to put forward recommendations to support healthcare decision-makers in integrating AI into the HTA processes. The barriers, addressed by the paper, are particularly focusing on Central and Eastern European (CEE) countries, where the implementation of HTA and access to health databases lag behind Western European countries. Methods We constructed a survey to rank the barriers to using AI for HTA purposes, completed by respondents from CEE jurisdictions with expertise in HTA. Using the results, two members of the HTx consortium from CEE developed recommendations on the most critical barriers. Then these recommendations were discussed in a workshop by a wider group of experts, including HTA and reimbursement decision-makers from both CEE countries and Western European countries, and summarized in a consensus report. Results Recommendations have been developed to address the top 15 barriers in areas of (1) human factor-related barriers, focusing on educating HTA doers and users, establishing collaborations and best practice sharing; (2) regulatory and policy-related barriers, proposing increasing awareness and political commitment and improving the management of sensitive information for AI use; (3) data-related barriers, suggesting enhancing standardization and collaboration with data networks, managing missing and unstructured data, using analytical and statistical approaches to address bias, using quality assessment tools and quality standards, improving reporting, and developing better conditions for the use of data; and (4) technological barriers, suggesting sustainable development of AI infrastructure. Conclusion In the field of HTA, the great potential of AI to support evidence generation and evaluation has not yet been sufficiently explored and realized. Raising awareness of the intended and unintended consequences of AI-based methods and encouraging political commitment from policymakers is necessary to upgrade the regulatory and infrastructural environment and knowledge base required to integrate AI into HTA-based decision-making processes better.
Collapse
Affiliation(s)
- Antal Zemplényi
- Center for Health Technology Assessment and Pharmacoeconomics Research, Faculty of Pharmacy, University of Pécs, Pécs, Hungary
- Syreon Research Institute, Budapest, Hungary
- *Correspondence: Antal Zemplényi,
| | - Konstantin Tachkov
- Department of Organization and Economics of Pharmacy, Faculty of Pharmacy, Medical University of Sofia, Sofia, Bulgaria
| | - Laszlo Balkanyi
- Medical Informatics R&D Center, Pannon University, Veszprém, Hungary
| | | | | | - Guenka Petrova
- Department of Organization and Economics of Pharmacy, Faculty of Pharmacy, Medical University of Sofia, Sofia, Bulgaria
| | - Marcin Czech
- Department of Pharmacoeconomics, Institute of Mother and Child, Warsaw, Poland
| | - Dalia Dawoud
- Science Policy and Research Programme, Science Evidence and Analytics Directorate, National Institute for Health and Care Excellence (NICE), London, United Kingdom
- Cairo University, Faculty of Pharmacy, Cairo, Egypt
| | - Wim Goettsch
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht University, Utrecht, Netherlands
- National Health Care Institute, Diemen, Netherlands
| | | | - Rok Hren
- Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia
| | - Saskia Knies
- National Health Care Institute, Diemen, Netherlands
| | - László Lorenzovici
- Syreon Research Romania, Tirgu Mures, Romania
- G. E. Palade University of Medicine, Pharmacy, Science and Technology, Tirgu Mures, Romania
| | | | - Oresta Piniazhko
- HTA Department of State Expert Centre of the Ministry of Health of Ukraine, Kyiv, Ukraine
| | - Alexandra Savova
- Department of Organization and Economics of Pharmacy, Faculty of Pharmacy, Medical University of Sofia, Sofia, Bulgaria
- National Council of Prices and Reimbursement of Medicinal Products, Sofia, Bulgaria
| | - Manoela Manova
- Department of Organization and Economics of Pharmacy, Faculty of Pharmacy, Medical University of Sofia, Sofia, Bulgaria
- National Council of Prices and Reimbursement of Medicinal Products, Sofia, Bulgaria
| | - Tomas Tesar
- Department of Organisation and Management of Pharmacy, Faculty of Pharmacy, Comenius University in Bratislava, Bratislava, Slovakia
| | | | - Zoltán Kaló
- Syreon Research Institute, Budapest, Hungary
- Centre for Health Technology Assessment, Semmelweis University, Budapest, Hungary
| |
Collapse
|
16
|
Al Meslamani AZ. Beyond implementation: the long-term economic impact of AI in healthcare. J Med Econ 2023; 26:1566-1569. [PMID: 37975706 DOI: 10.1080/13696998.2023.2285186] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Accepted: 11/15/2023] [Indexed: 11/19/2023]
Affiliation(s)
- Ahmad Z Al Meslamani
- College of Pharmacy, Al Ain University, Abu Dhabi, UAE
- AAU Health and Biomedical Research Center, Al Ain University, Abu Dhabi, UAE
| |
Collapse
|
17
|
Whicher D, Rapp T. The Value of Artificial Intelligence for Healthcare Decision Making-Lessons Learned. VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2022; 25:328-330. [PMID: 35227442 DOI: 10.1016/j.jval.2021.12.009] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/29/2021] [Accepted: 12/29/2021] [Indexed: 06/14/2023]
Affiliation(s)
| | - Thomas Rapp
- University of Paris, Paris, France; Sciences Po, LIEPP, Paris, France
| |
Collapse
|
18
|
Hashiguchi TCO, Oderkirk J, Slawomirski L. Fulfilling the Promise of Artificial Intelligence in the Health Sector: Let's Get Real. VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2022; 25:368-373. [PMID: 35227447 DOI: 10.1016/j.jval.2021.11.1369] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Revised: 11/17/2021] [Accepted: 11/23/2021] [Indexed: 06/14/2023]
Abstract
OBJECTIVES This study aimed to showcase the potential and key concerns and risks of artificial intelligence (AI) in the health sector, illustrating its application with current examples, and to provide policy guidance for the development, assessment, and adoption of AI technologies to advance policy objectives. METHODS Nonsystematic scan and analysis of peer-reviewed and gray literature on AI in the health sector, focusing on key insights for policy and governance. RESULTS The application of AI in the health sector is currently in the early stages. Most applications have not been scaled beyond the research setting. The use in real-world clinical settings is especially nascent, with more evidence in public health, biomedical research, and "back office" administration. Deploying AI in the health sector carries risks and hazards that must be managed proactively by policy makers. For AI to produce positive health and policy outcomes, 5 key areas for policy are proposed, including health data governance, operationalizing AI principles, flexible regulation, skills among health workers and patients, and strategic public investment. CONCLUSIONS AI is not a panacea, but a tool to address specific problems. Its successful development and adoption require data governance that ensures high-quality data are available and secure; relevant actors can access technical infrastructure and resources; regulatory frameworks promote trustworthy AI products; and health workers and patients have the information and skills to use AI products and services safely, effectively, and efficiently. All of this requires considerable investment and international collaboration.
Collapse
Affiliation(s)
- Tiago Cravo Oliveira Hashiguchi
- Organisation for Economic Co-operation and Development, Directorate for Employment Labour and Social Affairs, Paris, France.
| | - Jillian Oderkirk
- Organisation for Economic Co-operation and Development, Directorate for Employment Labour and Social Affairs, Paris, France
| | - Luke Slawomirski
- Organisation for Economic Co-operation and Development, Directorate for Employment Labour and Social Affairs, Paris, France
| |
Collapse
|
19
|
Crossnohere NL, Elsaid M, Paskett J, Bose-Brill S, Bridges JFP. Guidelines for artificial intelligence in medicine: A literature review and content analysis of frameworks (Preprint). J Med Internet Res 2022; 24:e36823. [PMID: 36006692 PMCID: PMC9459836 DOI: 10.2196/36823] [Citation(s) in RCA: 20] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2022] [Revised: 06/02/2022] [Accepted: 07/14/2022] [Indexed: 12/15/2022] Open
Abstract
Background Artificial intelligence (AI) is rapidly expanding in medicine despite a lack of consensus on its application and evaluation. Objective We sought to identify current frameworks guiding the application and evaluation of AI for predictive analytics in medicine and to describe the content of these frameworks. We also assessed what stages along the AI translational spectrum (ie, AI development, reporting, evaluation, implementation, and surveillance) the content of each framework has been discussed. Methods We performed a literature review of frameworks regarding the oversight of AI in medicine. The search included key topics such as “artificial intelligence,” “machine learning,” “guidance as topic,” and “translational science,” and spanned the time period 2014-2022. Documents were included if they provided generalizable guidance regarding the use or evaluation of AI in medicine. Included frameworks are summarized descriptively and were subjected to content analysis. A novel evaluation matrix was developed and applied to appraise the frameworks’ coverage of content areas across translational stages. Results Fourteen frameworks are featured in the review, including six frameworks that provide descriptive guidance and eight that provide reporting checklists for medical applications of AI. Content analysis revealed five considerations related to the oversight of AI in medicine across frameworks: transparency, reproducibility, ethics, effectiveness, and engagement. All frameworks include discussions regarding transparency, reproducibility, ethics, and effectiveness, while only half of the frameworks discuss engagement. The evaluation matrix revealed that frameworks were most likely to report AI considerations for the translational stage of development and were least likely to report considerations for the translational stage of surveillance. Conclusions Existing frameworks for the application and evaluation of AI in medicine notably offer less input on the role of engagement in oversight and regarding the translational stage of surveillance. Identifying and optimizing strategies for engagement are essential to ensure that AI can meaningfully benefit patients and other end users.
Collapse
Affiliation(s)
- Norah L Crossnohere
- Department of Biomedical Informatics, The Ohio State University College of Medicine, Columbus, OH, United States
- Division of General Internal Medicine, Department of Internal Medicine, The Ohio State University College of Medicine, Columbus, OH, United States
| | - Mohamed Elsaid
- Department of Biomedical Informatics, The Ohio State University College of Medicine, Columbus, OH, United States
| | - Jonathan Paskett
- Department of Biomedical Informatics, The Ohio State University College of Medicine, Columbus, OH, United States
| | - Seuli Bose-Brill
- Division of General Internal Medicine, Department of Internal Medicine, The Ohio State University College of Medicine, Columbus, OH, United States
| | - John F P Bridges
- Department of Biomedical Informatics, The Ohio State University College of Medicine, Columbus, OH, United States
| |
Collapse
|