1
|
Kim C, Gadgil SU, DeGrave AJ, Omiye JA, Cai ZR, Daneshjou R, Lee SI. Transparent medical image AI via an image-text foundation model grounded in medical literature. Nat Med 2024; 30:1154-1165. [PMID: 38627560 DOI: 10.1038/s41591-024-02887-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Accepted: 02/27/2024] [Indexed: 04/21/2024]
Abstract
Building trustworthy and transparent image-based medical artificial intelligence (AI) systems requires the ability to interrogate data and models at all stages of the development pipeline, from training models to post-deployment monitoring. Ideally, the data and associated AI systems could be described using terms already familiar to physicians, but this requires medical datasets densely annotated with semantically meaningful concepts. In the present study, we present a foundation model approach, named MONET (medical concept retriever), which learns how to connect medical images with text and densely scores images on concept presence to enable important tasks in medical AI development and deployment such as data auditing, model auditing and model interpretation. Dermatology provides a demanding use case for the versatility of MONET, due to the heterogeneity in diseases, skin tones and imaging modalities. We trained MONET based on 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature. MONET can accurately annotate concepts across dermatology images as verified by board-certified dermatologists, competitively with supervised models built on previously concept-annotated dermatology datasets of clinical images. We demonstrate how MONET enables AI transparency across the entire AI system development pipeline, from building inherently interpretable models to dataset and model auditing, including a case study dissecting the results of an AI clinical trial.
Collapse
Affiliation(s)
- Chanwoo Kim
- Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA
| | - Soham U Gadgil
- Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA
| | - Alex J DeGrave
- Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA
- Medical Scientist Training Program, University of Washington, Seattle, WA, USA
| | - Jesutofunmi A Omiye
- Department of Dermatology, Stanford School of Medicine, Stanford, CA, USA
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA
| | - Zhuo Ran Cai
- Program for Clinical Research and Technology, Stanford University, Stanford, CA, USA
| | - Roxana Daneshjou
- Department of Dermatology, Stanford School of Medicine, Stanford, CA, USA.
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA.
| | - Su-In Lee
- Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA.
| |
Collapse
|
3
|
Daneshjou R, Vodrahalli K, Novoa RA, Jenkins M, Liang W, Rotemberg V, Ko J, Swetter SM, Bailey EE, Gevaert O, Mukherjee P, Phung M, Yekrang K, Fong B, Sahasrabudhe R, Allerup JAC, Okata-Karigane U, Zou J, Chiou AS. Disparities in dermatology AI performance on a diverse, curated clinical image set. SCIENCE ADVANCES 2022; 8:eabq6147. [PMID: 35960806 PMCID: PMC9374341 DOI: 10.1126/sciadv.abq6147] [Citation(s) in RCA: 56] [Impact Index Per Article: 28.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 06/30/2022] [Indexed: 06/10/2023]
Abstract
An estimated 3 billion people lack access to dermatological care globally. Artificial intelligence (AI) may aid in triaging skin diseases and identifying malignancies. However, most AI models have not been assessed on images of diverse skin tones or uncommon diseases. Thus, we created the Diverse Dermatology Images (DDI) dataset-the first publicly available, expertly curated, and pathologically confirmed image dataset with diverse skin tones. We show that state-of-the-art dermatology AI models exhibit substantial limitations on the DDI dataset, particularly on dark skin tones and uncommon diseases. We find that dermatologists, who often label AI datasets, also perform worse on images of dark skin tones and uncommon diseases. Fine-tuning AI models on the DDI images closes the performance gap between light and dark skin tones. These findings identify important weaknesses and biases in dermatology AI that should be addressed for reliable application to diverse patients and diseases.
Collapse
Affiliation(s)
- Roxana Daneshjou
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA
| | - Kailas Vodrahalli
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
| | - Roberto A. Novoa
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
- Department of Pathology, Stanford School of Medicine, Stanford, CA, USA
| | - Melissa Jenkins
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Weixin Liang
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Veronica Rotemberg
- Dermatology Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Justin Ko
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Susan M. Swetter
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Elizabeth E. Bailey
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Olivier Gevaert
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA
| | - Pritam Mukherjee
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA
| | - Michelle Phung
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Kiana Yekrang
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Bradley Fong
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Rachna Sahasrabudhe
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | - Johan A. C. Allerup
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| | | | - James Zou
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Department of Computer Science, Stanford University, Stanford, CA, USA
- Chan-Zuckerberg Biohub, San Francisco, CA, USA
| | - Albert S. Chiou
- Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA
| |
Collapse
|
4
|
Advanced basal cell carcinoma: What dermatologists need to know about diagnosis. J Am Acad Dermatol 2022; 86:S1-S13. [PMID: 35577405 DOI: 10.1016/j.jaad.2022.03.023] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2021] [Revised: 03/07/2022] [Accepted: 03/09/2022] [Indexed: 12/07/2022]
Abstract
Basal cell carcinoma (BCC) is the most common human cancer, with approximately 3.6 million cases diagnosed each year. About 2000 deaths annually in the United States are attributed to basal and squamous cell skin cancers. There is a direct link between ultraviolet exposure and the development of BCC, as UV exposure damages DNA and induces mutations in tumor suppressor genes. Aberrations in the hedgehog pathway can also result in BCC, highlighted by the fact that most cases of sporadic BCCs have been found to have mutations in different genes involved in the hedgehog pathway. There are several genetic syndromes that are associated with BCCs, including basal cell nevus syndrome, xeroderma pigmentosum, Bazex-Dupré-Christol syndrome, Rombo syndrome, and Oley syndrome. Other risk factors include age, male gender, occupational hazards, radiation, and immunosuppression. BCCs are not typically staged but are instead stratified by their risk of recurring or metastasizing. Locally advanced BCCs are those tumors that are not amenable to surgery or radiation therapy.
Collapse
|