1
|
Cushnan D, Young KC, Ward D, Halling-Brown MD, Duffy S, Given-Wilson R, Wallis MG, Wilkinson L, Lyburn I, Sidebottom R, McAvinchey R, Lewis EB, Mackenzie A, Warren LM. Lessons learned from independent external validation of an AI tool to detect breast cancer using a representative UK data set. Br J Radiol 2023; 96:20211104. [PMID: 36607283 PMCID: PMC9975375 DOI: 10.1259/bjr.20211104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2021] [Revised: 11/21/2022] [Accepted: 11/30/2022] [Indexed: 01/07/2023] Open
Abstract
OBJECTIVE To pilot a process for the independent external validation of an artificial intelligence (AI) tool to detect breast cancer using data from the NHS breast screening programme (NHSBSP). METHODS A representative data set of mammography images from 26,000 women attending 2 NHS screening centres, and an enriched data set of 2054 positive cases were used from the OPTIMAM image database. The use case of the AI tool was the replacement of the first or second human reader. The performance of the AI tool was compared to that of human readers in the NHSBSP. RESULTS Recommendations for future external validations of AI tools to detect breast cancer are provided. The tool recalled different breast cancers to the human readers. This study showed the importance of testing AI tools on all types of cases (including non-standard) and the clarity of any warning messages. The acceptable difference in sensitivity and specificity between the AI tool and human readers should be determined. Any information vital for the clinical application should be a required output for the AI tool. It is recommended that the interaction of radiologists with the AI tool, and the effect of the AI tool on arbitration be investigated prior to clinical use. CONCLUSION This pilot demonstrated several lessons for future independent external validation of AI tools for breast cancer detection. ADVANCES IN KNOWLEDGE Knowledge has been gained towards best practice procedures for performing independent external validations of AI tools for the detection of breast cancer using data from the NHS Breast Screening Programme.
Collapse
Affiliation(s)
| | | | - Dominic Ward
- Royal Surrey NHS Foundation Trust, Guildford, United Kingdom
| | | | - Stephen Duffy
- Queen Mary University London, London, United Kingdom
| | | | - Matthew G Wallis
- Cambridge Breast Unit and NIHR Cambridge Biomedical Research Centre, Cambridge University Hospitals NHS Trust, Cambridge, United Kingdom
| | - Louise Wilkinson
- Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom
| | | | | | | | - Emma B Lewis
- Royal Surrey NHS Foundation Trust, Guildford, United Kingdom
| | | | - Lucy M Warren
- Royal Surrey NHS Foundation Trust, Guildford, United Kingdom
| |
Collapse
|
2
|
Cushnan D, Berka R, Bertolli O, Williams P, Schofield D, Joshi I, Favaro A, Halling-Brown M, Imreh G, Jefferson E, Sebire NJ, Reilly G, Rodrigues JCL, Robinson G, Copley S, Malik R, Bloomfield C, Gleeson F, Crotty M, Denton E, Dickson J, Leeming G, Hardwick HE, Baillie K, Openshaw PJ, Semple MG, Rubin C, Howlett A, Rockall AG, Bhayat A, Fascia D, Sudlow C, Jacob J. Towards nationally curated data archives for clinical radiology image analysis at scale: Learnings from national data collection in response to a pandemic. Digit Health 2021; 7:20552076211048654. [PMID: 34868617 PMCID: PMC8637703 DOI: 10.1177/20552076211048654] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2021] [Accepted: 09/07/2021] [Indexed: 12/27/2022] Open
Abstract
The prevalence of the coronavirus SARS-CoV-2 disease has resulted in the
unprecedented collection of health data to support research. Historically,
coordinating the collation of such datasets on a national scale has been
challenging to execute for several reasons, including issues with data privacy,
the lack of data reporting standards, interoperable technologies, and
distribution methods. The coronavirus SARS-CoV-2 disease pandemic has
highlighted the importance of collaboration between government bodies,
healthcare institutions, academic researchers and commercial companies in
overcoming these issues during times of urgency. The National COVID-19 Chest
Imaging Database, led by NHSX, British Society of Thoracic Imaging, Royal Surrey
NHS Foundation Trust and Faculty, is an example of such a national initiative.
Here, we summarise the experiences and challenges of setting up the National
COVID-19 Chest Imaging Database, and the implications for future ambitions of
national data curation in medical imaging to advance the safe adoption of
artificial intelligence in healthcare.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | - Mark Halling-Brown
- Scientific Computing, Royal Surrey NHS Foundation Trust, UK.,CVSSP, University of Surrey, UK
| | | | - Emily Jefferson
- Health Data Research UK, UK.,Health Informatics Centre (HIC), School of Medicine, University of Dundee, UK
| | | | | | | | - Graham Robinson
- Department of Radiology, Royal United Hospitals Bath NHS Foundation Trust, UK
| | - Susan Copley
- Imaging Department, Hammersmith Hospital, Imperial College NHS Healthcare Trust, UK
| | - Rizwan Malik
- Department of Radiology, Bolton NHS Foundation Trust, UK
| | - Claire Bloomfield
- National Consortium of Intelligent Medical Imaging (NCIMI), The Big Data Institute, University of Oxford, UK.,Dept of Oncology, University of Oxford, UK
| | - Fergus Gleeson
- National Consortium of Intelligent Medical Imaging (NCIMI), The Big Data Institute, University of Oxford, UK.,Dept of Oncology, University of Oxford, UK
| | | | - Erika Denton
- Norfolk and Norwich University Hospital Foundation Trust, UK
| | | | - Gary Leeming
- Institute of Population Health, Faculty of Health and Life Sciences, University of Liverpool, UK
| | - Hayley E Hardwick
- National Institute of Health Research (NIHR) Health Protection Research Unit in Emerging and Zoonotic Infections, UK
| | | | | | - Malcolm G Semple
- NIHR Health Protection Research Unit, Institute of Infection, Veterinary and Ecological Sciences, Faculty of Health and Life Sciences, University of Liverpool, UK
| | - Caroline Rubin
- Department of Radiology, University Hospital Southampton NHS Foundation Trust, UK
| | | | - Andrea G Rockall
- Imaging Department, Hammersmith Hospital, Imperial College NHS Healthcare Trust, UK.,Department of Surgery and Cancer, Faculty of Medicine, Imperial College London, UK
| | - Ayub Bhayat
- NHS Arden & Greater East Midlands Commissioning Support Unit, UK
| | | | - Cathie Sudlow
- British Heart Foundation Data Science Centre Led by Health Data Research UK, UK
| | | | - Joseph Jacob
- Department of Respiratory Medicine, University College London, UK.,Centre for Medical Image Computing, Department of Computer Science, University College London, UK
| |
Collapse
|
3
|
Cushnan D, Bennett O, Berka R, Bertolli O, Chopra A, Dorgham S, Favaro A, Ganepola T, Halling-Brown M, Imreh G, Jacob J, Jefferson E, Lemarchand F, Schofield D, Wyatt JC, Collaborative NCCID. Erratum to: An overview of the National COVID-19 Chest Imaging Database: data quality and cohort analysis. Gigascience 2021; 10:giab083. [PMID: 34850874 PMCID: PMC8634578 DOI: 10.1093/gigascience/giab083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Affiliation(s)
- Dominic Cushnan
- AI Lab, NHSX, Skipton House, 80 London Road, London SE1 6LH, UK
| | | | | | | | | | | | | | | | - Mark Halling-Brown
- Scientific Computing, Royal Surrey NHS Foundation Trust, Egerton Road, Guildford GU2 7XX, UK
| | | | - Joseph Jacob
- UCL Respiratory, 1st Floor, Rayne Institute, University College London, London WC1E 6JF, UK
| | - Emily Jefferson
- Health Data Research UK, Gibbs Building, 215 Euston Road, London NW1 2BE, UK
- Health Informatics Centre (HIC), School of Medicine, University of Dundee, DD1 4HN, Dundee, UK
| | | | | | - Jeremy C Wyatt
- Emeritus Professor of Digital Healthcare, University of Southampton, Southampton SO17 1BJ, UK
- NHSX, Skipton House, 80 London Road, London SE1 6LH, UK
| | | |
Collapse
|
4
|
Cushnan D, Bennett O, Berka R, Bertolli O, Chopra A, Dorgham S, Favaro A, Ganepola T, Halling-Brown M, Imreh G, Jacob J, Jefferson E, Lemarchand F, Schofield D, Wyatt JC. An overview of the National COVID-19 Chest Imaging Database: data quality and cohort analysis. Gigascience 2021; 10:giab076. [PMID: 34849869 PMCID: PMC8633457 DOI: 10.1093/gigascience/giab076] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2021] [Revised: 08/04/2021] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND The National COVID-19 Chest Imaging Database (NCCID) is a centralized database containing mainly chest X-rays and computed tomography scans from patients across the UK. The objective of the initiative is to support a better understanding of the coronavirus SARS-CoV-2 disease (COVID-19) and the development of machine learning technologies that will improve care for patients hospitalized with a severe COVID-19 infection. This article introduces the training dataset, including a snapshot analysis covering the completeness of clinical data, and availability of image data for the various use-cases (diagnosis, prognosis, longitudinal risk). An additional cohort analysis measures how well the NCCID represents the wider COVID-19-affected UK population in terms of geographic, demographic, and temporal coverage. FINDINGS The NCCID offers high-quality DICOM images acquired across a variety of imaging machinery; multiple time points including historical images are available for a subset of patients. This volume and variety make the database well suited to development of diagnostic/prognostic models for COVID-associated respiratory conditions. Historical images and clinical data may aid long-term risk stratification, particularly as availability of comorbidity data increases through linkage to other resources. The cohort analysis revealed good alignment to general UK COVID-19 statistics for some categories, e.g., sex, whilst identifying areas for improvements to data collection methods, particularly geographic coverage. CONCLUSION The NCCID is a growing resource that provides researchers with a large, high-quality database that can be leveraged both to support the response to the COVID-19 pandemic and as a test bed for building clinically viable medical imaging models.
Collapse
Affiliation(s)
- Dominic Cushnan
- AI Lab, NHSX, Skipton House, 80 London Road, London SE1 6LH,
UK
| | | | | | | | | | | | | | | | - Mark Halling-Brown
- Scientific Computing, Royal Surrey NHS Foundation Trust,
Egerton Road, Guildford GU2 7XX, UK
| | | | - Joseph Jacob
- UCL Respiratory, 1st Floor, Rayne Institute, University College
London, London WC1E 6JF, UK
| | - Emily Jefferson
- Health Data Research UK, Gibbs Building, 215 Euston Road,
London NW1 2BE, UK
- Health Informatics Centre (HIC), School of Medicine, University of
Dundee, DD1 4HN, Dundee, UK
| | | | | | - Jeremy C Wyatt
- Emeritus Professor of Digital Healthcare, University of
Southampton, Southampton SO17 1BJ, UK
- NHSX, Skipton House, 80 London Road, London SE1 6LH, UK
| | | |
Collapse
|
5
|
Sounderajah V, Ashrafian H, Rose S, Shah NH, Ghassemi M, Golub R, Kahn CE, Esteva A, Karthikesalingam A, Mateen B, Webster D, Milea D, Ting D, Treanor D, Cushnan D, King D, McPherson D, Glocker B, Greaves F, Harling L, Ordish J, Cohen JF, Deeks J, Leeflang M, Diamond M, McInnes MDF, McCradden M, Abràmoff MD, Normahani P, Markar SR, Chang S, Liu X, Mallett S, Shetty S, Denniston A, Collins GS, Moher D, Whiting P, Bossuyt PM, Darzi A. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med 2021; 27:1663-1665. [PMID: 34635854 DOI: 10.1038/s41591-021-01517-0] [Citation(s) in RCA: 58] [Impact Index Per Article: 19.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Viknesh Sounderajah
- Institute of Global Health Innovation, Imperial College London, London, UK.
- Department of Surgery and Cancer, Imperial College London, London, UK.
| | - Hutan Ashrafian
- Institute of Global Health Innovation, Imperial College London, London, UK
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Sherri Rose
- Center for Health Policy and Center for Primary Care and Outcomes Research, Stanford University, Stanford, CA, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| | - Marzyeh Ghassemi
- Institute for Medical Engineering & Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Robert Golub
- Journal of the American Medical Association (JAMA), Chicago, IL, USA
| | - Charles E Kahn
- University of Pennsylvania, Philadelphia, Pennsylvania, PA, USA
| | | | | | | | | | - Dan Milea
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
| | - Daniel Ting
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
| | - Darren Treanor
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
- University of Leeds, Leeds, UK
- Department of Clinical Pathology, and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden
- Center for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| | | | - Dominic King
- Institute of Global Health Innovation, Imperial College London, London, UK
- Optum, London, UK
| | | | - Ben Glocker
- Faculty of Engineering, Department of Computing, Imperial College London, London, UK
| | - Felix Greaves
- National Institute for Health and Care Excellence, London, UK
| | - Leanne Harling
- Institute of Global Health Innovation, Imperial College London, London, UK
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Johan Ordish
- Medicines and Healthcare Products Regulatory Agency, London, UK
| | - Jérémie F Cohen
- Department of Pediatrics, Centre of Research in Epidemiology and Statistics, Inserm UMR 1153, Necker- Enfants Malades Hospital, Assistance Publique-Hôpitaux de Paris, Université de Paris, Paris, France
| | - Jon Deeks
- Institute of Applied Health Research, University of Birmingham, Birmingham, UK
| | - Mariska Leeflang
- Department of Epidemiology and Data Science, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, The Netherlands
| | | | - Matthew D F McInnes
- Departments of Radiology and Epidemiology, University of Ottawa, The Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Melissa McCradden
- Department of Bioethics, The Hospital for Sick Kids, Toronto, Ontario, Canada
| | - Michael D Abràmoff
- Department of Ophthalmology and Visual Sciences, University of Iowa, Iowa City, IA, USA
| | - Pasha Normahani
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Sheraz R Markar
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Stephanie Chang
- Annals of Internal Medicine, American College of Physicians, Philadelphia, PA, USA
| | - Xiaoxuan Liu
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK, London, UK
| | - Susan Mallett
- Centre for Medical Imaging, University College London, London, UK
| | | | - Alastair Denniston
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK, London, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - David Moher
- Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Penny Whiting
- Bristol Medical School, University of Bristol, Bristol, UK
| | - Patrick M Bossuyt
- Department of Epidemiology and Data Science, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, The Netherlands.
| | - Ara Darzi
- Institute of Global Health Innovation, Imperial College London, London, UK.
- Department of Surgery and Cancer, Imperial College London, London, UK.
| |
Collapse
|
6
|
Jacob J, Alexander D, Baillie JK, Berka R, Bertolli O, Blackwood J, Buchan I, Bloomfield C, Cushnan D, Docherty A, Edey A, Favaro A, Gleeson F, Halling-Brown M, Hare S, Jefferson E, Johnstone A, Kirby M, McStay R, Nair A, Openshaw PJM, Parker G, Reilly G, Robinson G, Roditi G, Rodrigues JCL, Sebire N, Semple MG, Sudlow C, Woznitza N, Joshi I. Using imaging to combat a pandemic: rationale for developing the UK National COVID-19 Chest Imaging Database. Eur Respir J 2020; 56:2001809. [PMID: 32616598 PMCID: PMC7331656 DOI: 10.1183/13993003.01809-2020] [Citation(s) in RCA: 19] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2020] [Accepted: 06/08/2020] [Indexed: 12/12/2022]
Abstract
The National COVID-19 Chest Imaging Database (NCCID) is a repository of chest radiographs, CT and MRI images and clinical data from COVID-19 patients across the UK, to support research and development of AI technology and give insight into COVID-19 disease https://bit.ly/3eQeuha
Collapse
Affiliation(s)
- Joseph Jacob
- Dept of Respiratory Medicine, University College London, London, UK
- Centre for Medical Image Computing, Dept of Computer Science, University College London, London, UK
| | - Daniel Alexander
- Centre for Medical Image Computing, Dept of Computer Science, University College London, London, UK
| | - J Kenneth Baillie
- Division of Genetics and Genomics, The Roslin Institute, University of Edinburgh, Edinburgh, UK
- Centre for Inflammation Research, University of Edinburgh, Edinburgh, UK
| | | | | | - James Blackwood
- The Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD), Dept of eHealth, NHS Greater Glasgow and Clyde, Glasgow, UK
| | - Iain Buchan
- Institute of Population Health, University of Liverpool, Liverpool, UK
| | - Claire Bloomfield
- National Consortium of Intelligent Medical Imaging (NCIMI), The University of Oxford, Big Data Institute, Oxford, UK
| | | | - Annemarie Docherty
- Centre for Medical Informatics, Usher Institute, University of Edinburgh, Edinburgh, UK
| | - Anthony Edey
- Dept of Radiology, Southmead Hospital, North Bristol NHS Trust, Bristol, UK
| | | | - Fergus Gleeson
- National Consortium of Intelligent Medical Imaging (NCIMI), The University of Oxford, Big Data Institute, Oxford, UK
- Dept of Oncology, University of Oxford, Oxford, UK
| | - Mark Halling-Brown
- Scientific Computing, Royal Surrey NHS Foundation Trust, Guildford, UK
- Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, UK
| | - Samanjit Hare
- Dept of Radiology, Royal Free London NHS Trust, London, UK
| | - Emily Jefferson
- Health Data Research UK, London, UK
- Health Informatics Centre (HIC), School of Medicine, University of Dundee, Dundee, UK
| | - Annette Johnstone
- Dept of Radiology, Leeds Teaching Hospitals NHS Trust, Leeds General Infirmary, Leeds, UK
| | | | - Ruth McStay
- Dept of Radiology, Freeman Hospital, Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, UK
| | - Arjun Nair
- Dept of Radiology, University College London Hospital, London, UK
| | - Peter J M Openshaw
- National Heart and Lung Institute, Faculty of Medicine, Imperial College London, London, UK
| | - Geoff Parker
- Centre for Medical Image Computing, Dept of Computer Science, University College London, London, UK
- Bioxydyn Limited, Manchester, UK
| | | | - Graham Robinson
- Dept of Radiology, Royal United Hospitals Bath NHS Foundation Trust, Bath, UK
| | - Giles Roditi
- Dept of Radiology, University of Glasgow, Glasgow Royal Infirmary, Glasgow, UK
| | | | | | - Malcolm G Semple
- NIHR Health Protection Research Unit in Emerging and Zoonotic Infections, Faculty of Health and Life Sciences, University of Liverpool, Liverpool, UK
| | - Catherine Sudlow
- Usher Institute, University of Edinburgh, Edinburgh, UK
- British Heart Foundation (BHF) Data Science Centre, Health Data Research UK, Edinburgh, UK
| | - Nick Woznitza
- Radiology Dept, Homerton University Hospital, London, UK
- School of Allied and Public Health Professions, Canterbury Christ Church University, Canterbury, UK
- 12 NHS Nightingale Hospital London, London, UK
| | | |
Collapse
|