1
|
Souza R, Wilms M, Camacho M, Pike GB, Camicioli R, Monchi O, Forkert ND. Image-encoded biological and non-biological variables may be used as shortcuts in deep learning models trained on multisite neuroimaging data. J Am Med Inform Assoc 2023; 30:1925-1933. [PMID: 37669158 PMCID: PMC10654841 DOI: 10.1093/jamia/ocad171] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Revised: 08/07/2023] [Accepted: 08/15/2023] [Indexed: 09/07/2023] Open
Abstract
OBJECTIVE This work investigates if deep learning (DL) models can classify originating site locations directly from magnetic resonance imaging (MRI) scans with and without correction for intensity differences. MATERIAL AND METHODS A large database of 1880 T1-weighted MRI scans collected across 41 sites originally for Parkinson's disease (PD) classification was used to classify sites in this study. Forty-six percent of the datasets are from PD patients, while 54% are from healthy participants. After preprocessing the T1-weighted scans, 2 additional data types were generated: intensity-harmonized T1-weighted scans and log-Jacobian deformation maps resulting from nonlinear atlas registration. Corresponding DL models were trained to classify sites for each data type. Additionally, logistic regression models were used to investigate the contribution of biological (age, sex, disease status) and non-biological (scanner type) variables to the models' decision. RESULTS A comparison of the 3 different types of data revealed that DL models trained using T1-weighted and intensity-harmonized T1-weighted scans can classify sites with an accuracy of 85%, while the model using log-Jacobian deformation maps achieved a site classification accuracy of 54%. Disease status and scanner type were found to be significant confounders. DISCUSSION Our results demonstrate that MRI scans encode relevant site-specific information that models could use as shortcuts that cannot be removed using simple intensity harmonization methods. CONCLUSION The ability of DL models to exploit site-specific biases as shortcuts raises concerns about their reliability, generalization, and deployability in clinical settings.
Collapse
Affiliation(s)
- Raissa Souza
- Department of Radiology, Cumming School of Medicine, University of Calgary, Calgary, AB T2N 4N1, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
- Biomedical Engineering Graduate Program, University of Calgary, Calgary, AB T2N 4N1, Canada
| | - Matthias Wilms
- Hotchkiss Brain Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
- Alberta Children’s Hospital Research Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
- Department of Pediatrics, University of Calgary, Calgary, AB T2N 4N1, Canada
- Department of Community Health Sciences, University of Calgary, Calgary, AB T2N 4N1, Canada
| | - Milton Camacho
- Department of Radiology, Cumming School of Medicine, University of Calgary, Calgary, AB T2N 4N1, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
- Biomedical Engineering Graduate Program, University of Calgary, Calgary, AB T2N 4N1, Canada
| | - G Bruce Pike
- Department of Radiology, Cumming School of Medicine, University of Calgary, Calgary, AB T2N 4N1, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
| | - Richard Camicioli
- Department of Medicine (Neurology), Neuroscience and Mental Health Institute, University of Alberta, Edmonton, AB T6G 2E1, Canada
| | - Oury Monchi
- Hotchkiss Brain Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
- Department of Radiology, Radio-Oncology and Nuclear Medicine, Université de Montréal, Montréal, QC H3C 3J7, Canada
- Centre de Recherche, Institut Universitaire de Gériatrie de Montréal, Montréal, QC H3W 1W4, Canada
- Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB T2N 4N1, Canada
| | - Nils D Forkert
- Department of Radiology, Cumming School of Medicine, University of Calgary, Calgary, AB T2N 4N1, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
- Alberta Children’s Hospital Research Institute, University of Calgary, Calgary, AB T2N 4N1, Canada
- Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB T2N 4N1, Canada
| |
Collapse
|