1
|
Goessinger EV, Cerminara SE, Mueller AM, Gottfrois P, Huber S, Amaral M, Wenz F, Kostner L, Weiss L, Kunz M, Maul JT, Wespi S, Broman E, Kaufmann S, Patpanathapillai V, Treyer I, Navarini AA, Maul LV. Consistency of convolutional neural networks in dermoscopic melanoma recognition: A prospective real-world study about the pitfalls of augmented intelligence. J Eur Acad Dermatol Venereol 2024; 38:945-953. [PMID: 38158385 DOI: 10.1111/jdv.19777] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2023] [Accepted: 10/23/2023] [Indexed: 01/03/2024]
Abstract
BACKGROUND Deep-learning convolutional neural networks (CNNs) have outperformed even experienced dermatologists in dermoscopic melanoma detection under controlled conditions. It remains unexplored how real-world dermoscopic image transformations affect CNN robustness. OBJECTIVES To investigate the consistency of melanoma risk assessment by two commercially available CNNs to help formulate recommendations for current clinical use. METHODS A comparative cohort study was conducted from January to July 2022 at the Department of Dermatology, University Hospital Basel. Five dermoscopic images of 116 different lesions on the torso of 66 patients were captured consecutively by the same operator without deliberate rotation. Classification was performed by two CNNs (CNN-1/CNN-2). Lesions were divided into four subgroups based on their initial risk scoring and clinical dignity assessment. Reliability was assessed by variation and intraclass correlation coefficients. Excisions were performed for melanoma suspicion or two consecutively elevated CNN risk scores, and benign lesions were confirmed by expert consensus (n = 3). RESULTS 117 repeated image series of 116 melanocytic lesions (2 melanomas, 16 dysplastic naevi, 29 naevi, 1 solar lentigo, 1 suspicious and 67 benign) were classified. CNN-1 demonstrated superior measurement repeatability for clinically benign lesions with an initial malignant risk score (mean variation coefficient (mvc): CNN-1: 49.5(±34.3)%; CNN-2: 71.4(±22.5)%; p = 0.03), while CNN-2 outperformed for clinically benign lesions with benign scoring (mvc: CNN-1: 49.7(±22.7)%; CNN-2: 23.8(±29.3)%; p = 0.002). Both systems exhibited lowest score consistency for lesions with an initial malignant risk score and benign assessment. In this context, averaging three initial risk scores achieved highest sensitivity of dignity assessment (CNN-1: 94%; CNN-2: 89%). Intraclass correlation coefficients indicated 'moderate'-to-'good' reliability for both systems (CNN-1: 0.80, 95% CI:0.71-0.87, p < 0.001; CNN-2: 0.67, 95% CI:0.55-0.77, p < 0.001). CONCLUSIONS Potential user-induced image changes can significantly influence CNN classification. For clinical application, we recommend using the average of three initial risk scores. Furthermore, we advocate for CNN robustness optimization by cross-validation with repeated image sets. TRIAL REGISTRATION ClinicalTrials.gov (NCT04605822).
Collapse
Affiliation(s)
- E V Goessinger
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - S E Cerminara
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - A M Mueller
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - P Gottfrois
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - S Huber
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - M Amaral
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - F Wenz
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - L Kostner
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - L Weiss
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - M Kunz
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - J-T Maul
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - S Wespi
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - E Broman
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - S Kaufmann
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - V Patpanathapillai
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - I Treyer
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - A A Navarini
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
| | - L V Maul
- Department of Dermatology, University Hospital Basel, Basel, Switzerland
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| |
Collapse
|
2
|
Amruthalingam L, Mang N, Gottfrois P, Jimenez AG, Maul J, Kunz M, Pouly M, Navarini A. 143 Automated Anatomical Mapping of Hand Eczema. J Invest Dermatol 2022. [DOI: 10.1016/j.jid.2022.09.153] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
3
|
Amruthalingam L, Gottfrois P, Gonzalez Jimenez A, Gökduman B, Kunz M, Koller T, Pouly M, Navarini A. Improved diagnosis by automated macro- and micro-anatomical region mapping of skin photographs. J Eur Acad Dermatol Venereol 2022; 36:2525-2532. [PMID: 35924423 PMCID: PMC9804282 DOI: 10.1111/jdv.18476] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Accepted: 06/30/2022] [Indexed: 01/05/2023]
Abstract
BACKGROUND The exact location of skin lesions is key in clinical dermatology. On one hand, it supports differential diagnosis (DD) since most skin conditions have specific predilection sites. On the other hand, location matters for dermatosurgical interventions. In practice, lesion evaluation is not well standardized and anatomical descriptions vary or lack altogether. Automated determination of anatomical location could benefit both situations. OBJECTIVE Establish an automated method to determine anatomical regions in clinical patient pictures and evaluate the gain in DD performance of a deep learning model (DLM) when trained with lesion locations and images. METHODS Retrospective study based on three datasets: macro-anatomy for the main body regions with 6000 patient pictures partially labelled by a student, micro-anatomy for the ear region with 182 pictures labelled by a student and DD with 3347 pictures of 16 diseases determined by dermatologists in clinical settings. For each dataset, a DLM was trained and evaluated on an independent test set. The primary outcome measures were the precision and sensitivity with 95% CI. For DD, we compared the performance of a DLM trained with lesion pictures only with a DLM trained with both pictures and locations. RESULTS The average precision and sensitivity were 85% (CI 84-86), 84% (CI 83-85) for macro-anatomy, 81% (CI 80-83), 80% (CI 77-83) for micro-anatomy and 82% (CI 78-85), 81% (CI 77-84) for DD. We observed an improvement in DD performance of 6% (McNemar test P-value 0.0009) for both average precision and sensitivity when training with both lesion pictures and locations. CONCLUSION Including location can be beneficial for DD DLM performance. The proposed method can generate body region maps from patient pictures and even reach surgery relevant anatomical precision, e.g. the ear region. Our method enables automated search of large clinical databases and make targeted anatomical image retrieval possible.
Collapse
Affiliation(s)
- L. Amruthalingam
- Department of Biomedical EngineeringUniversity of BaselBaselSwitzerland,Lucerne School of Computer Science and Information TechnologyLucerne University of Applied Sciences and ArtsLucerneSwitzerland
| | - P. Gottfrois
- Department of Biomedical EngineeringUniversity of BaselBaselSwitzerland
| | | | - B. Gökduman
- Lucerne School of Computer Science and Information TechnologyLucerne University of Applied Sciences and ArtsLucerneSwitzerland
| | - M. Kunz
- Department of Health Sciences and TechnologySwiss Federal Institute of TechnologyZurichSwitzerland
| | - T. Koller
- Department of DermatologyUniversity Hospital of BaselBaselSwitzerland
| | | | - M. Pouly
- Department of DermatologyUniversity Hospital of BaselBaselSwitzerland
| | - A.A. Navarini
- Department of Health Sciences and TechnologySwiss Federal Institute of TechnologyZurichSwitzerland
| |
Collapse
|