1
|
Nazari S, Garcia R. Automatic Skin Cancer Detection Using Clinical Images: A Comprehensive Review. Life (Basel) 2023; 13:2123. [PMID: 38004263 PMCID: PMC10672549 DOI: 10.3390/life13112123] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2023] [Revised: 10/21/2023] [Accepted: 10/23/2023] [Indexed: 11/26/2023] Open
Abstract
Skin cancer has become increasingly common over the past decade, with melanoma being the most aggressive type. Hence, early detection of skin cancer and melanoma is essential in dermatology. Computational methods can be a valuable tool for assisting dermatologists in identifying skin cancer. Most research in machine learning for skin cancer detection has focused on dermoscopy images due to the existence of larger image datasets. However, general practitioners typically do not have access to a dermoscope and must rely on naked-eye examinations or standard clinical images. By using standard, off-the-shelf cameras to detect high-risk moles, machine learning has also proven to be an effective tool. The objective of this paper is to provide a comprehensive review of image-processing techniques for skin cancer detection using clinical images. In this study, we evaluate 51 state-of-the-art articles that have used machine learning methods to detect skin cancer over the past decade, focusing on clinical datasets. Even though several studies have been conducted in this field, there are still few publicly available clinical datasets with sufficient data that can be used as a benchmark, especially when compared to the existing dermoscopy databases. In addition, we observed that the available artifact removal approaches are not quite adequate in some cases and may also have a negative impact on the models. Moreover, the majority of the reviewed articles are working with single-lesion images and do not consider typical mole patterns and temporal changes in the lesions of each patient.
Collapse
|
2
|
Lim K, Neal-Smith G, Mitchell C, Xerri J, Chuanromanee P. Perceptions of the use of artificial intelligence in the diagnosis of skin cancer: an outpatient survey. Clin Exp Dermatol 2021; 47:542-546. [PMID: 34610153 DOI: 10.1111/ced.14969] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2020] [Revised: 05/04/2021] [Accepted: 10/04/2021] [Indexed: 11/30/2022]
Abstract
BACKGROUND Convolutional neural networks (artificial intelligence, AI) are rapidly appearing within the field of dermatology, with diagnostic accuracy matching that of dermatologists. As technologies become available for use by both the health professionals and the general public, their uptake in healthcare will become more acceptable. National Health Service England recognizes the potential of AI for healthcare but emphasizes that patient-centred care should be at the forefront of these technological advancements. AIM To obtain opinions of patients on the use of AI in a dermatology setting, when aiding the diagnosis of skin cancers. METHODS A cross-sectional 14-point questionnaire was handed out to patients attending dermatology outpatient skin cancer clinics in two UK hospitals, between March and August 2018. RESULTS In total, 603 patient questionnaires were completed. Nearly half (47%; n = 282) of respondents were not concerned if AI technology was used by a skin specialist to aid skin cancer diagnosis. However, the majority (81%; n = 491) of respondents, considered it important for a dermatologist to examine and confirm a diagnosis and to be present for discussion of a cancer diagnosis. CONCLUSION Although the majority of respondents were not reluctant about the use of AI for skin cancer diagnosis, respondents still considered it important that dermatologists are involved in the diagnosis and/or confirmation of skin cancer. Furthermore, the study results demonstrate that personal interaction with a clinician is important. This is in keeping with proposals that AI be used as an adjunctive technology to increase accuracy of skin cancer diagnoses, but not as a substitute for a dermatologist.
Collapse
Affiliation(s)
- K Lim
- Department of Dermatology, Oxford University NHS Hospitals Trust, Oxford, UK
| | - G Neal-Smith
- Department of Dermatology, Oxford University NHS Hospitals Trust, Oxford, UK
| | - C Mitchell
- Department of Dermatology, St Mary's Hospital, Portsmouth NHS Hospital Trust, Portsmouth, UK
| | - J Xerri
- Department of Statistics, University of Oxford, Oxford, UK
| | - P Chuanromanee
- Department of Statistics, University of Oxford, Oxford, UK
| |
Collapse
|
3
|
Sies K, Winkler JK, Fink C, Bardehle F, Toberer F, Buhl T, Enk A, Blum A, Rosenberger A, Haenssle HA. Past and present of computer-assisted dermoscopic diagnosis: performance of a conventional image analyser versus a convolutional neural network in a prospective data set of 1,981 skin lesions. Eur J Cancer 2020; 135:39-46. [PMID: 32534243 DOI: 10.1016/j.ejca.2020.04.043] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2019] [Accepted: 04/29/2020] [Indexed: 01/10/2023]
Abstract
BACKGROUND Convolutional neural networks (CNNs) have shown a dermatologist-level performance in the classification of skin lesions. We aimed to deliver a head-to-head comparison of a conventional image analyser (CIA), which depends on segmentation and weighting of handcrafted features, to a CNN trained by deep learning. METHODS Cross-sectional study using a real-world, prospectively acquired, dermoscopic dataset of 1981 skin lesions to compare the diagnostic performance of a market-approved CNN (Moleanalyzer-Pro™, developed in 2018) to a CIA (Moleanalyzer-3™/Dynamole™; developed in 2004, all FotoFinder Systems Inc, Germany). As a reference standard, we used histopathological diagnoses (n = 785) or, in non-excised benign lesions (n = 1196), expert consensus plus an uneventful follow-up by sequential digital dermoscopy for at least 2 years. RESULTS A total of 281 malignant lesions and 1700 benign lesions from 435 patients (62.2% male, mean age: 52 years) were prospectively imaged. The CNN showed a sensitivity of 77.6% (95% confidence interval [CI]: [72.4%-82.1%]), specificity of 95.3% (95% CI: [94.2%-96.2%]), and receiver operating characteristic (ROC)-area under the curve (AUC) of 0.945 (95% CI: [0.930-0.961]). In contrast, the CIA achieved a sensitivity of 53.4% (95% CI: [47.5%-59.1%]), specificity of 86.6% (95% CI: [84.9%-88.1%]) and ROC-AUC of 0.738 (95% CI: [0.701-0.774]). The data set included melanomas originally diagnosed by dynamic changes during sequential digital dermoscopy (52 of 201, 20.6%), which reduced the sensitivities of both classifiers. Pairwise comparisons of sensitivities, specificities, and ROC-AUCs indicated a clear outperformance by the CNN (all p < 0.001). CONCLUSIONS The superior diagnostic performance of the CNN argues against a continued application of former CIAs as an aide to physicians' clinical management decisions.
Collapse
Affiliation(s)
- Katharina Sies
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Julia K Winkler
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Christine Fink
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Felicitas Bardehle
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Ferdinand Toberer
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Timo Buhl
- Department of Dermatology, University of Göttingen, Göttingen, Germany
| | - Alexander Enk
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Andreas Blum
- Office Based Clinic of Dermatology, Konstanz, Germany
| | - Albert Rosenberger
- Department of Genetic Epidemiology, University of Goettingen, Goettingen, Germany
| | - Holger A Haenssle
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany.
| |
Collapse
|
4
|
Ferrante di Ruffano L, Takwoingi Y, Dinnes J, Chuchu N, Bayliss SE, Davenport C, Matin RN, Godfrey K, O'Sullivan C, Gulati A, Chan SA, Durack A, O'Connell S, Gardiner MD, Bamber J, Deeks JJ, Williams HC. Computer-assisted diagnosis techniques (dermoscopy and spectroscopy-based) for diagnosing skin cancer in adults. Cochrane Database Syst Rev 2018; 12:CD013186. [PMID: 30521691 PMCID: PMC6517147 DOI: 10.1002/14651858.cd013186] [Citation(s) in RCA: 44] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
BACKGROUND Early accurate detection of all skin cancer types is essential to guide appropriate management and to improve morbidity and survival. Melanoma and cutaneous squamous cell carcinoma (cSCC) are high-risk skin cancers which have the potential to metastasise and ultimately lead to death, whereas basal cell carcinoma (BCC) is usually localised with potential to infiltrate and damage surrounding tissue. Anxiety around missing early curable cases needs to be balanced against inappropriate referral and unnecessary excision of benign lesions. Computer-assisted diagnosis (CAD) systems use artificial intelligence to analyse lesion data and arrive at a diagnosis of skin cancer. When used in unreferred settings ('primary care'), CAD may assist general practitioners (GPs) or other clinicians to more appropriately triage high-risk lesions to secondary care. Used alongside clinical and dermoscopic suspicion of malignancy, CAD may reduce unnecessary excisions without missing melanoma cases. OBJECTIVES To determine the accuracy of CAD systems for diagnosing cutaneous invasive melanoma and atypical intraepidermal melanocytic variants, BCC or cSCC in adults, and to compare its accuracy with that of dermoscopy. SEARCH METHODS We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials (CENTRAL); MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles. SELECTION CRITERIA Studies of any design that evaluated CAD alone, or in comparison with dermoscopy, in adults with lesions suspicious for melanoma or BCC or cSCC, and compared with a reference standard of either histological confirmation or clinical follow-up. DATA COLLECTION AND ANALYSIS Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). We contacted authors of included studies where information related to the target condition or diagnostic threshold were missing. We estimated summary sensitivities and specificities separately by type of CAD system, using the bivariate hierarchical model. We compared CAD with dermoscopy using (a) all available CAD data (indirect comparisons), and (b) studies providing paired data for both tests (direct comparisons). We tested the contribution of human decision-making to the accuracy of CAD diagnoses in a sensitivity analysis by removing studies that gave CAD results to clinicians to guide diagnostic decision-making. MAIN RESULTS We included 42 studies, 24 evaluating digital dermoscopy-based CAD systems (Derm-CAD) in 23 study cohorts with 9602 lesions (1220 melanomas, at least 83 BCCs, 9 cSCCs), providing 32 datasets for Derm-CAD and seven for dermoscopy. Eighteen studies evaluated spectroscopy-based CAD (Spectro-CAD) in 16 study cohorts with 6336 lesions (934 melanomas, 163 BCC, 49 cSCCs), providing 32 datasets for Spectro-CAD and six for dermoscopy. These consisted of 15 studies using multispectral imaging (MSI), two studies using electrical impedance spectroscopy (EIS) and one study using diffuse-reflectance spectroscopy. Studies were incompletely reported and at unclear to high risk of bias across all domains. Included studies inadequately address the review question, due to an abundance of low-quality studies, poor reporting, and recruitment of highly selected groups of participants.Across all CAD systems, we found considerable variation in the hardware and software technologies used, the types of classification algorithm employed, methods used to train the algorithms, and which lesion morphological features were extracted and analysed across all CAD systems, and even between studies evaluating CAD systems. Meta-analysis found CAD systems had high sensitivity for correct identification of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants in highly selected populations, but with low and very variable specificity, particularly for Spectro-CAD systems. Pooled data from 22 studies estimated the sensitivity of Derm-CAD for the detection of melanoma as 90.1% (95% confidence interval (CI) 84.0% to 94.0%) and specificity as 74.3% (95% CI 63.6% to 82.7%). Pooled data from eight studies estimated the sensitivity of multispectral imaging CAD (MSI-CAD) as 92.9% (95% CI 83.7% to 97.1%) and specificity as 43.6% (95% CI 24.8% to 64.5%). When applied to a hypothetical population of 1000 lesions at the mean observed melanoma prevalence of 20%, Derm-CAD would miss 20 melanomas and would lead to 206 false-positive results for melanoma. MSI-CAD would miss 14 melanomas and would lead to 451 false diagnoses for melanoma. Preliminary findings suggest CAD systems are at least as sensitive as assessment of dermoscopic images for the diagnosis of invasive melanoma and atypical intraepidermal melanocytic variants. We are unable to make summary statements about the use of CAD in unreferred populations, or its accuracy in detecting keratinocyte cancers, or its use in any setting as a diagnostic aid, because of the paucity of studies. AUTHORS' CONCLUSIONS In highly selected patient populations all CAD types demonstrate high sensitivity, and could prove useful as a back-up for specialist diagnosis to assist in minimising the risk of missing melanomas. However, the evidence base is currently too poor to understand whether CAD system outputs translate to different clinical decision-making in practice. Insufficient data are available on the use of CAD in community settings, or for the detection of keratinocyte cancers. The evidence base for individual systems is too limited to draw conclusions on which might be preferred for practice. Prospective comparative studies are required that evaluate the use of already evaluated CAD systems as diagnostic aids, by comparison to face-to-face dermoscopy, and in participant populations that are representative of those in which the test would be used in practice.
Collapse
Affiliation(s)
| | - Yemisi Takwoingi
- University of BirminghamInstitute of Applied Health ResearchEdgbaston CampusBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Jacqueline Dinnes
- University of BirminghamInstitute of Applied Health ResearchEdgbaston CampusBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Naomi Chuchu
- University of BirminghamInstitute of Applied Health ResearchEdgbaston CampusBirminghamUKB15 2TT
| | - Susan E Bayliss
- University of BirminghamInstitute of Applied Health ResearchEdgbaston CampusBirminghamUKB15 2TT
| | - Clare Davenport
- University of BirminghamInstitute of Applied Health ResearchEdgbaston CampusBirminghamUKB15 2TT
| | - Rubeta N Matin
- Churchill HospitalDepartment of DermatologyOld RoadHeadingtonOxfordUKOX3 7LE
| | - Kathie Godfrey
- The University of Nottinghamc/o Cochrane Skin GroupNottinghamUK
| | | | - Abha Gulati
- Barts Health NHS TrustDepartment of DermatologyWhitechapelLondonUKE11BB
| | - Sue Ann Chan
- City HospitalBirmingham Skin CentreDudley RdBirminghamUKB18 7QH
| | - Alana Durack
- Addenbrooke’s Hospital, Cambridge University Hospitals NHS Foundation TrustDermatologyHills RoadCambridgeUKCB2 0QQ
| | - Susan O'Connell
- Cardiff and Vale University Health BoardCEDAR Healthcare Technology Research CentreCardiff Medicentre, University Hospital of Wales, Heath Park CampusCardiffWalesUKCF144UJ
| | | | - Jeffrey Bamber
- Institute of Cancer Research and The Royal Marsden NHS Foundation TrustJoint Department of Physics15 Cotswold RoadSuttonUKSM2 5NG
| | - Jonathan J Deeks
- University of BirminghamInstitute of Applied Health ResearchEdgbaston CampusBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Hywel C Williams
- University of NottinghamCentre of Evidence Based DermatologyQueen's Medical CentreDerby RoadNottinghamUKNG7 2UH
| | | | | |
Collapse
|
5
|
Dinnes J, Deeks JJ, Chuchu N, Matin RN, Wong KY, Aldridge RB, Durack A, Gulati A, Chan SA, Johnston L, Bayliss SE, Leonardi‐Bee J, Takwoingi Y, Davenport C, O'Sullivan C, Tehrani H, Williams HC. Visual inspection and dermoscopy, alone or in combination, for diagnosing keratinocyte skin cancers in adults. Cochrane Database Syst Rev 2018; 12:CD011901. [PMID: 30521688 PMCID: PMC6516870 DOI: 10.1002/14651858.cd011901.pub2] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
BACKGROUND Early accurate detection of all skin cancer types is important to guide appropriate management, to reduce morbidity and to improve survival. Basal cell carcinoma (BCC) is almost always a localised skin cancer with potential to infiltrate and damage surrounding tissue, whereas a minority of cutaneous squamous cell carcinomas (cSCCs) and invasive melanomas are higher-risk skin cancers with the potential to metastasise and cause death. Dermoscopy has become an important tool to assist specialist clinicians in the diagnosis of melanoma, and is increasingly used in primary-care settings. Dermoscopy is a precision-built handheld illuminated magnifier that allows more detailed examination of the skin down to the level of the superficial dermis. Establishing the value of dermoscopy over and above visual inspection for the diagnosis of BCC or cSCC in primary- and secondary-care settings is critical to understanding its potential contribution to appropriate skin cancer triage, including referral of higher-risk cancers to secondary care, the identification of low-risk skin cancers that might be treated in primary care and to provide reassurance to those with benign skin lesions who can be safely discharged. OBJECTIVES To determine the diagnostic accuracy of visual inspection and dermoscopy, alone or in combination, for the detection of (a) BCC and (b) cSCC, in adults. We separated studies according to whether the diagnosis was recorded face-to-face (in person) or based on remote (image-based) assessment. SEARCH METHODS We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles. SELECTION CRITERIA Studies of any design that evaluated visual inspection or dermoscopy or both in adults with lesions suspicious for skin cancer, compared with a reference standard of either histological confirmation or clinical follow-up. DATA COLLECTION AND ANALYSIS Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). We contacted authors of included studies where information related to the target condition or diagnostic thresholds were missing. We estimated accuracy using hierarchical summary ROC methods. We undertook analysis of studies allowing direct comparison between tests. To facilitate interpretation of results, we computed values of sensitivity at the point on the SROC curve with 80% fixed specificity and values of specificity with 80% fixed sensitivity. We investigated the impact of in-person test interpretation; use of a purposely-developed algorithm to assist diagnosis; and observer expertise. MAIN RESULTS We included 24 publications reporting on 24 study cohorts, providing 27 visual inspection datasets (8805 lesions; 2579 malignancies) and 33 dermoscopy datasets (6855 lesions; 1444 malignancies). The risk of bias was mainly low for the index test (for dermoscopy evaluations) and reference standard domains, particularly for in-person evaluations, and high or unclear for participant selection, application of the index test for visual inspection and for participant flow and timing. We scored concerns about the applicability of study findings as of 'high' or 'unclear' concern for almost all studies across all domains assessed. Selective participant recruitment, lack of reproducibility of diagnostic thresholds and lack of detail on observer expertise were particularly problematic.The detection of BCC was reported in 28 datasets; 15 on an in-person basis and 13 image-based. Analysis of studies by prior testing of participants and according to observer expertise was not possible due to lack of data. Studies were primarily conducted in participants referred for specialist assessment of lesions with available histological classification. We found no clear differences in accuracy between dermoscopy studies undertaken in person and those which evaluated images. The lack of effect observed may be due to other sources of heterogeneity, including variations in the types of skin lesion studied, in dermatoscopes used, or in the use of algorithms and varying thresholds for deciding on a positive test result.Meta-analysis found in-person evaluations of dermoscopy (7 evaluations; 4683 lesions and 363 BCCs) to be more accurate than visual inspection alone for the detection of BCC (8 evaluations; 7017 lesions and 1586 BCCs), with a relative diagnostic odds ratio (RDOR) of 8.2 (95% confidence interval (CI) 3.5 to 19.3; P < 0.001). This corresponds to predicted differences in sensitivity of 14% (93% versus 79%) at a fixed specificity of 80% and predicted differences in specificity of 22% (99% versus 77%) at a fixed sensitivity of 80%. We observed very similar results for the image-based evaluations.When applied to a hypothetical population of 1000 lesions, of which 170 are BCC (based on median BCC prevalence across studies), an increased sensitivity of 14% from dermoscopy would lead to 24 fewer BCCs missed, assuming 166 false positive results from both tests. A 22% increase in specificity from dermoscopy with sensitivity fixed at 80% would result in 183 fewer unnecessary excisions, assuming 34 BCCs missed for both tests. There was not enough evidence to assess the use of algorithms or structured checklists for either visual inspection or dermoscopy.Insufficient data were available to draw conclusions on the accuracy of either test for the detection of cSCCs. AUTHORS' CONCLUSIONS Dermoscopy may be a valuable tool for the diagnosis of BCC as an adjunct to visual inspection of a suspicious skin lesion following a thorough history-taking including assessment of risk factors for keratinocyte cancer. The evidence primarily comes from secondary-care (referred) populations and populations with pigmented lesions or mixed lesion types. There is no clear evidence supporting the use of currently-available formal algorithms to assist dermoscopy diagnosis.
Collapse
Affiliation(s)
- Jacqueline Dinnes
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Jonathan J Deeks
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Naomi Chuchu
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
| | - Rubeta N Matin
- Churchill HospitalDepartment of DermatologyOld RoadHeadingtonOxfordUKOX3 7LE
| | - Kai Yuen Wong
- Oxford University Hospitals NHS Foundation TrustDepartment of Plastic and Reconstructive SurgeryOxfordUK
| | - Roger Benjamin Aldridge
- NHS Lothian/University of EdinburghDepartment of Plastic Surgery25/6 India StreetEdinburghUKEH3 6HE
| | - Alana Durack
- Addenbrooke’s Hospital, Cambridge University Hospitals NHS Foundation TrustDermatologyHills RoadCambridgeUKCB2 0QQ
| | - Abha Gulati
- Barts Health NHS TrustDepartment of DermatologyWhitechapelLondonUKE11BB
| | - Sue Ann Chan
- City HospitalBirmingham Skin CentreDudley RdBirminghamUKB18 7QH
| | - Louise Johnston
- NIHR Diagnostic Evidence Co‐operative Newcastle2nd Floor William Leech Building (Rm M2.061) Institute of Cellular Medicine Newcastle UniversityFramlington PlaceNewcastle upon TyneUKNE2 4HH
| | - Susan E Bayliss
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
| | - Jo Leonardi‐Bee
- The University of NottinghamDivision of Epidemiology and Public HealthClinical Sciences BuildingNottingham City Hospital NHS Trust Campus, Hucknall RoadNottinghamUKNG5 1PB
| | - Yemisi Takwoingi
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Clare Davenport
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
| | | | - Hamid Tehrani
- Whiston HospitalDepartment of Plastic and Reconstructive SurgeryWarrington RoadLiverpoolUKL35 5DR
| | - Hywel C Williams
- University of NottinghamCentre of Evidence Based DermatologyQueen's Medical CentreDerby RoadNottinghamUKNG7 2UH
| | | | | |
Collapse
|
6
|
Dinnes J, Deeks JJ, Chuchu N, Ferrante di Ruffano L, Matin RN, Thomson DR, Wong KY, Aldridge RB, Abbott R, Fawzy M, Bayliss SE, Grainge MJ, Takwoingi Y, Davenport C, Godfrey K, Walter FM, Williams HC. Dermoscopy, with and without visual inspection, for diagnosing melanoma in adults. Cochrane Database Syst Rev 2018; 12:CD011902. [PMID: 30521682 PMCID: PMC6517096 DOI: 10.1002/14651858.cd011902.pub2] [Citation(s) in RCA: 62] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/13/2023]
Abstract
BACKGROUND Melanoma has one of the fastest rising incidence rates of any cancer. It accounts for a small percentage of skin cancer cases but is responsible for the majority of skin cancer deaths. Although history-taking and visual inspection of a suspicious lesion by a clinician are usually the first in a series of 'tests' to diagnose skin cancer, dermoscopy has become an important tool to assist diagnosis by specialist clinicians and is increasingly used in primary care settings. Dermoscopy is a magnification technique using visible light that allows more detailed examination of the skin compared to examination by the naked eye alone. Establishing the additive value of dermoscopy over and above visual inspection alone across a range of observers and settings is critical to understanding its contribution for the diagnosis of melanoma and to future understanding of the potential role of the growing number of other high-resolution image analysis techniques. OBJECTIVES To determine the diagnostic accuracy of dermoscopy alone, or when added to visual inspection of a skin lesion, for the detection of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants in adults. We separated studies according to whether the diagnosis was recorded face-to-face (in-person), or based on remote (image-based), assessment. SEARCH METHODS We undertook a comprehensive search of the following databases from inception up to August 2016: CENTRAL; MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles. SELECTION CRITERIA Studies of any design that evaluated dermoscopy in adults with lesions suspicious for melanoma, compared with a reference standard of either histological confirmation or clinical follow-up. Data on the accuracy of visual inspection, to allow comparisons of tests, was included only if reported in the included studies of dermoscopy. DATA COLLECTION AND ANALYSIS Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). We contacted authors of included studies where information related to the target condition or diagnostic threshold were missing. We estimated accuracy using hierarchical summary receiver operating characteristic (SROC),methods. Analysis of studies allowing direct comparison between tests was undertaken. To facilitate interpretation of results, we computed values of sensitivity at the point on the SROC curve with 80% fixed specificity and values of specificity with 80% fixed sensitivity. We investigated the impact of in-person test interpretation; use of a purposely developed algorithm to assist diagnosis; observer expertise; and dermoscopy training. MAIN RESULTS We included a total of 104 study publications reporting on 103 study cohorts with 42,788 lesions (including 5700 cases), providing 354 datasets for dermoscopy. The risk of bias was mainly low for the index test and reference standard domains and mainly high or unclear for participant selection and participant flow. Concerns regarding the applicability of study findings were largely scored as 'high' concern in three of four domains assessed. Selective participant recruitment, lack of reproducibility of diagnostic thresholds and lack of detail on observer expertise were particularly problematic.The accuracy of dermoscopy for the detection of invasive melanoma or atypical intraepidermal melanocytic variants was reported in 86 datasets; 26 for evaluations conducted in person (dermoscopy added to visual inspection), and 60 for image-based evaluations (diagnosis based on interpretation of dermoscopic images). Analyses of studies by prior testing revealed no obvious effect on accuracy; analyses were hampered by the lack of studies in primary care, lack of relevant information and the restricted inclusion of lesions selected for biopsy or excision. Accuracy was higher for in-person diagnosis compared to image-based evaluations (relative diagnostic odds ratio (RDOR) 4.6, 95% confidence interval (CI) 2.4 to 9.0; P < 0.001).We compared accuracy for (a), in-person evaluations of dermoscopy (26 evaluations; 23,169 lesions and 1664 melanomas),versus visual inspection alone (13 evaluations; 6740 lesions and 459 melanomas), and for (b), image-based evaluations of dermoscopy (60 evaluations; 13,475 lesions and 2851 melanomas),versus image-based visual inspection (11 evaluations; 1740 lesions and 305 melanomas). For both comparisons, meta-analysis found dermoscopy to be more accurate than visual inspection alone, with RDORs of (a), 4.7 (95% CI 3.0 to 7.5; P < 0.001), and (b), 5.6 (95% CI 3.7 to 8.5; P < 0.001). For a), the predicted difference in sensitivity at a fixed specificity of 80% was 16% (95% CI 8% to 23%; 92% for dermoscopy + visual inspection versus 76% for visual inspection), and predicted difference in specificity at a fixed sensitivity of 80% was 20% (95% CI 7% to 33%; 95% for dermoscopy + visual inspection versus 75% for visual inspection). For b) the predicted differences in sensitivity was 34% (95% CI 24% to 46%; 81% for dermoscopy versus 47% for visual inspection), at a fixed specificity of 80%, and predicted difference in specificity was 40% (95% CI 27% to 57%; 82% for dermoscopy versus 42% for visual inspection), at a fixed sensitivity of 80%.Using the median prevalence of disease in each set of studies ((a), 12% for in-person and (b), 24% for image-based), for a hypothetical population of 1000 lesions, an increase in sensitivity of (a), 16% (in-person), and (b), 34% (image-based), from using dermoscopy at a fixed specificity of 80% equates to a reduction in the number of melanomas missed of (a), 19 and (b), 81 with (a), 176 and (b), 152 false positive results. An increase in specificity of (a), 20% (in-person), and (b), 40% (image-based), at a fixed sensitivity of 80% equates to a reduction in the number of unnecessary excisions from using dermoscopy of (a), 176 and (b), 304 with (a), 24 and (b), 48 melanomas missed.The use of a named or published algorithm to assist dermoscopy interpretation (as opposed to no reported algorithm or reported use of pattern analysis), had no significant impact on accuracy either for in-person (RDOR 1.4, 95% CI 0.34 to 5.6; P = 0.17), or image-based (RDOR 1.4, 95% CI 0.60 to 3.3; P = 0.22), evaluations. This result was supported by subgroup analysis according to algorithm used. We observed higher accuracy for observers reported as having high experience and for those classed as 'expert consultants' in comparison to those considered to have less experience in dermoscopy, particularly for image-based evaluations. Evidence for the effect of dermoscopy training on test accuracy was very limited but suggested associated improvements in sensitivity. AUTHORS' CONCLUSIONS Despite the observed limitations in the evidence base, dermoscopy is a valuable tool to support the visual inspection of a suspicious skin lesion for the detection of melanoma and atypical intraepidermal melanocytic variants, particularly in referred populations and in the hands of experienced users. Data to support its use in primary care are limited, however, it may assist in triaging suspicious lesions for urgent referral when employed by suitably trained clinicians. Formal algorithms may be of most use for dermoscopy training purposes and for less expert observers, however reliable data comparing approaches using dermoscopy in person are lacking.
Collapse
Affiliation(s)
- Jacqueline Dinnes
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Jonathan J Deeks
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Naomi Chuchu
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
| | | | - Rubeta N Matin
- Churchill HospitalDepartment of DermatologyOld RoadHeadingtonOxfordUKOX3 7LE
| | | | - Kai Yuen Wong
- Oxford University Hospitals NHS Foundation TrustDepartment of Plastic and Reconstructive SurgeryOxfordUK
| | - Roger Benjamin Aldridge
- NHS Lothian/University of EdinburghDepartment of Plastic Surgery25/6 India StreetEdinburghUKEH3 6HE
| | - Rachel Abbott
- University Hospital of WalesWelsh Institute of DermatologyHeath ParkCardiffUKCF14 4XW
| | - Monica Fawzy
- Norfolk and Norwich University Hospital NHS TrustDepartment of Plastic and Reconstructive SurgeryColney LaneNorwichUKNR4 7UY
| | - Susan E Bayliss
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
| | - Matthew J Grainge
- School of MedicineDivision of Epidemiology and Public HealthUniversity of NottinghamNottinghamUKNG7 2UH
| | - Yemisi Takwoingi
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
- University Hospitals Birmingham NHS Foundation Trust and University of BirminghamNIHR Birmingham Biomedical Research CentreBirminghamUK
| | - Clare Davenport
- University of BirminghamInstitute of Applied Health ResearchBirminghamUKB15 2TT
| | - Kathie Godfrey
- The University of Nottinghamc/o Cochrane Skin GroupNottinghamUK
| | - Fiona M Walter
- University of CambridgePublic Health & Primary CareStrangeways Research Laboratory, Worts CausewayCambridgeUKCB1 8RN
| | - Hywel C Williams
- University of NottinghamCentre of Evidence Based DermatologyQueen's Medical CentreDerby RoadNottinghamUKNG7 2UH
| | | | | |
Collapse
|
7
|
Fink C, Uhlmann L, Hofmann M, Forschner A, Eigentler T, Garbe C, Enk A, Haenssle HA. Patientenakzeptanz und -vertrauen in die automatisierte, computergestützte Diagnostik des Melanoms mithilfe der Dermatofluoroskopie. J Dtsch Dermatol Ges 2018; 16:854-860. [DOI: 10.1111/ddg.13562_g] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2017] [Accepted: 12/19/2017] [Indexed: 11/29/2022]
Affiliation(s)
- Christine Fink
- Department of Dermatology; University Medical Center, Ruprecht Karls University Heidelberg, Germany (Hautklinik, Universitätsklinikum Heidelberg, Ruprecht-Karls-Universität Heidelberg, Deutschland)
| | - Lorenz Uhlmann
- Institute of Medical Biometry and Informatics; University of Heidelberg, Heidelberg, Germany (Institut für medizinische Biometrie und Informatik, Universität Heidelberg, Heidelberg, Deutschland)
| | - Maja Hofmann
- University Department of Dermatology; Charité - Universitätsmedizin Berlin, Berlin, Germany (Klinik für Dermatologie, Venerologie und Allergologie, Charité - Universitätsmedizin Berlin, Berlin, Deutschland)
| | - Andrea Forschner
- Center of Dermato-oncology; Department of Dermatology; Eberhard Karls University, Tuebingen, Germany (Zentrum für Dermatoonkologie, Universitäts-Hautklinik Tübingen, Eberhard-Karls-Unversität, Tübingen, Deutschland)
| | - Thomas Eigentler
- Center of Dermato-oncology; Department of Dermatology; Eberhard Karls University, Tuebingen, Germany (Zentrum für Dermatoonkologie, Universitäts-Hautklinik Tübingen, Eberhard-Karls-Unversität, Tübingen, Deutschland)
| | - Claus Garbe
- Center of Dermato-oncology; Department of Dermatology; Eberhard Karls University, Tuebingen, Germany (Zentrum für Dermatoonkologie, Universitäts-Hautklinik Tübingen, Eberhard-Karls-Unversität, Tübingen, Deutschland)
| | - Alexander Enk
- Department of Dermatology; University Medical Center, Ruprecht Karls University Heidelberg, Germany (Hautklinik, Universitätsklinikum Heidelberg, Ruprecht-Karls-Universität Heidelberg, Deutschland)
| | - Holger A. Haenssle
- Department of Dermatology; University Medical Center, Ruprecht Karls University Heidelberg, Germany (Hautklinik, Universitätsklinikum Heidelberg, Ruprecht-Karls-Universität Heidelberg, Deutschland)
| |
Collapse
|
8
|
Stolz W. Wie schätzen unsere Patienten die Melanomfrüherkennung mit Hilfe des Computers und von hochentwickelten technischen Systemen ein? J Dtsch Dermatol Ges 2018; 16:819-820. [DOI: 10.1111/ddg.13567] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
9
|
Fink C, Uhlmann L, Hofmann M, Forschner A, Eigentler T, Garbe C, Enk A, Haenssle HA. Patient acceptance and trust in automated computer-assisted diagnosis of melanoma with dermatofluoroscopy. J Dtsch Dermatol Ges 2018; 16:854-859. [PMID: 29927518 DOI: 10.1111/ddg.13562] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2017] [Accepted: 12/19/2017] [Indexed: 01/13/2023]
Abstract
BACKGROUND AND OBJECTIVES Automated computer-guided diagnostic procedures are increasingly being integrated into patient care. However, in contrast to the increasing application of automation, patient acceptance and trust in such technologies has rarely been studied. Automated diagnosis of melanoma with dermatofluoroscopy was recently approved by regulatory agencies. The objective of this study is to assess patient acceptance and trust in automated melanoma diagnosis with dermatofluoroscopy. PATIENTS AND METHODS We examined 140 pigmented skin lesions with dermatofluoroscopy as part of a prospective clinical study. Four weeks after their examination with dermatofluoroscopy, we contacted 100 patients with a 10-item questionnaire addressing their acceptance and trust in this technology on a five-point visual analogue scale. RESULTS A "high" to "very high" level of patient acceptance and trust in dermatofluoroscopy was found in 74 % of responders. Most patients agreed that computer-assisted diagnoses are trustworthy and may generally improve the diagnostic performance of physicians. However, all responders insisted on the interpretation of computer-assisted diagnoses by a physician and frequently rejected the idea of computers completely replacing physicians. CONCLUSION Patient acceptance and trust in dermatofluoroscopy was high. Patients clearly supported the use of automated, computer-assisted diagnostics as an adjunct to the physicians' examination.
Collapse
Affiliation(s)
- Christine Fink
- Department of Dermatology, University Medical Center, Ruprecht Karl University, Heidelberg, Germany
| | - Lorenz Uhlmann
- Institute of Medical Biometry and Informatics, University of Heidelberg, Heidelberg, Germany
| | - Maja Hofmann
- University Department of Dermatology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Andrea Forschner
- Center of Dermato-oncology, Department of Dermatology, Eberhard Karls University, Tuebingen, Germany
| | - Thomas Eigentler
- Center of Dermato-oncology, Department of Dermatology, Eberhard Karls University, Tuebingen, Germany
| | - Claus Garbe
- Center of Dermato-oncology, Department of Dermatology, Eberhard Karls University, Tuebingen, Germany
| | - Alexander Enk
- Department of Dermatology, University Medical Center, Ruprecht Karl University, Heidelberg, Germany
| | - Holger A Haenssle
- Department of Dermatology, University Medical Center, Ruprecht Karl University, Heidelberg, Germany
| |
Collapse
|
10
|
Del Rosario F, Farahi JM, Drendel J, Buntinx-Krieg T, Caravaglio J, Domozych R, Chapman S, Braunberger T, Dellavalle RP, Norris DA, Fathi R, Alkousakis T. Performance of a computer-aided digital dermoscopic image analyzer for melanoma detection in 1,076 pigmented skin lesion biopsies. J Am Acad Dermatol 2018; 78:927-934.e6. [PMID: 29678380 DOI: 10.1016/j.jaad.2017.01.049] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2016] [Revised: 01/27/2017] [Accepted: 01/29/2017] [Indexed: 11/26/2022]
Abstract
BACKGROUND Digital dermoscopic image analysis of pigmented skin lesions (PSLs) has become increasingly popular, despite its unclear clinical utility. Unbiased, high-powered studies investigating the efficacy of commercially available systems are limited. OBJECTIVE To investigate the diagnostic performance of the FotoFinder Mole-Analyzer in assessing PSLs for cutaneous melanoma. METHODS In this 15-year retrospective study, the histopathologies of 1076 biopsied PSLs among a total of 2500 imaged PSLs were collected. The biopsied PSLs were categorized as benign or malignant (cutaneous melanoma) based on histopathology. Analyzer scores (0-1.00) for these PSLs were obtained and grouped according to histopathology. RESULTS At an optimized cutoff score of 0.50, a sensitivity of 56% and a specificity of 74% were achieved. The area under the receiver operating characteristics curve was 0.698, indicating poor accuracy as a diagnostic tool. LIMITATIONS This study had a retrospective design and involved only a single institution. CONCLUSION Our study reveals a low sensitivity of the scoring function of this digital dermoscopic image analyzer for detecting cutaneous melanomas. Physicians must apply keen clinical judgment when using such devices in the screening of suspicious PSLs.
Collapse
Affiliation(s)
| | | | | | | | | | - Renee Domozych
- University of Central Florida College of Medicine, Orlando, Florida
| | - Stephanie Chapman
- Michigan State University, College of Human Medicine, Grand Rapids, Michigan
| | - Taylor Braunberger
- University of North Dakota School of Medicine, Grand Forks, North Dakota
| | | | - David A Norris
- University of Colorado School of Medicine, Aurora, Colorado
| | - Ramin Fathi
- University of Colorado School of Medicine, Aurora, Colorado
| | | |
Collapse
|
11
|
Computer-Aided Decision Support for Melanoma Detection Applied on Melanocytic and Nonmelanocytic Skin Lesions: A Comparison of Two Systems Based on Automatic Analysis of Dermoscopic Images. BIOMED RESEARCH INTERNATIONAL 2015; 2015:579282. [PMID: 26693486 PMCID: PMC4674594 DOI: 10.1155/2015/579282] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 09/11/2015] [Accepted: 11/03/2015] [Indexed: 11/29/2022]
Abstract
Commercially available clinical decision support systems (CDSSs) for skin cancer have been designed for the detection of melanoma only. Correct use of the systems requires expert knowledge, hampering their utility for nonexperts. Furthermore, there are no systems to detect other common skin cancer types, that is, nonmelanoma skin cancer (NMSC). As early diagnosis of skin cancer is essential, there is a need for a CDSS that is applicable to all types of skin lesions and is suitable for nonexperts. Nevus Doctor (ND) is a CDSS being developed by the authors. We here investigate ND's ability to detect both melanoma and NMSC and the opportunities for improvement. An independent test set of dermoscopic images of 870 skin lesions, including 44 melanomas and 101 NMSCs, were analysed by ND. Its sensitivity to melanoma and NMSC was compared to that of Mole Expert (ME), a commercially available CDSS, using the same set of lesions. ND and ME had similar sensitivity to melanoma. For ND at 95% melanoma sensitivity, the NMSC sensitivity was 100%, and the specificity was 12%. The melanomas misclassified by ND at 95% sensitivity were correctly classified by ME, and vice versa. ND is able to detect NMSC without sacrificing melanoma sensitivity.
Collapse
|
12
|
Zortea M, Schopf TR, Thon K, Geilhufe M, Hindberg K, Kirchesch H, Møllersen K, Schulz J, Skrøvseth SO, Godtliebsen F. Performance of a dermoscopy-based computer vision system for the diagnosis of pigmented skin lesions compared with visual evaluation by experienced dermatologists. Artif Intell Med 2013; 60:13-26. [PMID: 24382424 DOI: 10.1016/j.artmed.2013.11.006] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2013] [Revised: 11/28/2013] [Accepted: 11/29/2013] [Indexed: 10/25/2022]
Abstract
BACKGROUND It is often difficult to differentiate early melanomas from benign melanocytic nevi even by expert dermatologists, and the task is even more challenging for primary care physicians untrained in dermatology and dermoscopy. A computer system can provide an objective and quantitative evaluation of skin lesions, reducing subjectivity in the diagnosis. OBJECTIVE Our objective is to make a low-cost computer aided diagnostic tool applicable in primary care based on a consumer grade camera with attached dermatoscope, and compare its performance to that of experienced dermatologists. METHODS AND MATERIALS We propose several new image-derived features computed from automatically segmented dermoscopic pictures. These are related to the asymmetry, color, border, geometry, and texture of skin lesions. The diagnostic accuracy of the system is compared with that of three dermatologists. RESULTS With a data set of 206 skin lesions, 169 benign and 37 melanomas, the classifier was able to provide competitive sensitivity (86%) and specificity (52%) scores compared with the sensitivity (85%) and specificity (48%) of the most accurate dermatologist using only dermoscopic images. CONCLUSION We show that simple statistical classifiers can be trained to provide a recommendation on whether a pigmented skin lesion requires biopsy to exclude skin cancer with a performance that is comparable to and exceeds that of experienced dermatologists.
Collapse
Affiliation(s)
- Maciel Zortea
- Department of Mathematics and Statistics, University of Tromsø, 9037 Tromsø, Norway.
| | - Thomas R Schopf
- Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, 9038 Tromsø, Norway
| | - Kevin Thon
- Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, 9038 Tromsø, Norway
| | - Marc Geilhufe
- Department of Mathematics and Statistics, University of Tromsø, 9037 Tromsø, Norway
| | - Kristian Hindberg
- Department of Mathematics and Statistics, University of Tromsø, 9037 Tromsø, Norway
| | | | - Kajsa Møllersen
- Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, 9038 Tromsø, Norway
| | - Jörn Schulz
- Department of Mathematics and Statistics, University of Tromsø, 9037 Tromsø, Norway
| | - Stein Olav Skrøvseth
- Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, 9038 Tromsø, Norway
| | - Fred Godtliebsen
- Department of Mathematics and Statistics, University of Tromsø, 9037 Tromsø, Norway
| |
Collapse
|
13
|
Computerized analysis of pigmented skin lesions: A review. Artif Intell Med 2012; 56:69-90. [DOI: 10.1016/j.artmed.2012.08.002] [Citation(s) in RCA: 238] [Impact Index Per Article: 19.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2012] [Revised: 08/02/2012] [Accepted: 08/19/2012] [Indexed: 11/20/2022]
|