1
|
Wei Y, Deng Y, Sun C, Lin M, Jiang H, Peng Y. Deep learning with noisy labels in medical prediction problems: a scoping review. J Am Med Inform Assoc 2024:ocae108. [PMID: 38814164 DOI: 10.1093/jamia/ocae108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Revised: 04/27/2024] [Accepted: 05/03/2024] [Indexed: 05/31/2024] Open
Abstract
OBJECTIVES Medical research faces substantial challenges from noisy labels attributed to factors like inter-expert variability and machine-extracted labels. Despite this, the adoption of label noise management remains limited, and label noise is largely ignored. To this end, there is a critical need to conduct a scoping review focusing on the problem space. This scoping review aims to comprehensively review label noise management in deep learning-based medical prediction problems, which includes label noise detection, label noise handling, and evaluation. Research involving label uncertainty is also included. METHODS Our scoping review follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched 4 databases, including PubMed, IEEE Xplore, Google Scholar, and Semantic Scholar. Our search terms include "noisy label AND medical/healthcare/clinical," "uncertainty AND medical/healthcare/clinical," and "noise AND medical/healthcare/clinical." RESULTS A total of 60 papers met inclusion criteria between 2016 and 2023. A series of practical questions in medical research are investigated. These include the sources of label noise, the impact of label noise, the detection of label noise, label noise handling techniques, and their evaluation. Categorization of both label noise detection methods and handling techniques are provided. DISCUSSION From a methodological perspective, we observe that the medical community has been up to date with the broader deep-learning community, given that most techniques have been evaluated on medical data. We recommend considering label noise as a standard element in medical research, even if it is not dedicated to handling noisy labels. Initial experiments can start with easy-to-implement methods, such as noise-robust loss functions, weighting, and curriculum learning.
Collapse
Affiliation(s)
- Yishu Wei
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY 10065, United States
- Reddit Inc., San Francisco, CA 16093, United States
| | - Yu Deng
- Center for Health Information Partnerships, Northwestern University, Chicago, IL 10611, United States
| | - Cong Sun
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY 10065, United States
| | - Mingquan Lin
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY 10065, United States
- Department of Surgery, University of Minnesota, Minneapolis, MN 55455, United States
| | - Hongmei Jiang
- Department of Statistics and Data Science, Northwestern University, Evanston, IL 60208, United States
| | - Yifan Peng
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY 10065, United States
| |
Collapse
|
2
|
Thykjaer AS, Andresen J, Andersen N, Bek T, Heegaard S, Hajari J, Schmidt Laugesen C, Möller S, Pedersen FN, Kawasaki R, Højlund K, Rubin KH, Stokholm L, Peto T, Grauslund J. Inter-grader reliability in the Danish screening programme for diabetic retinopathy. Acta Ophthalmol 2023; 101:783-788. [PMID: 37066883 DOI: 10.1111/aos.15667] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2022] [Revised: 11/18/2022] [Accepted: 03/27/2023] [Indexed: 04/18/2023]
Abstract
PURPOSE The Danish Registry of Diabetic Retinopathy includes information from >200 000 patients who attends diabetic retinopathy (DR) screening in Denmark. Screening of patients with uncomplicated type 2 diabetes is often performed by practicing ophthalmologists, while patients with type 1 and complicated type 2 diabetes attends screening at hospitals. We performed a clinical reliability study of retinal images from Danish screening facilities to explore the inter-grader agreement between the primary screening ophthalmologist and a blinded, certified grader. METHODS Invitations to participate were sent to screening facilities across Denmark. The primary grader uploaded fundus photographs with information on estimated level of DR (International Clinical Diabetic Retinopathy scale as 0 [no DR], 1-3 [mild, moderate or severe nonproliferative DR {NPDR}], or 4 [proliferative DR {PDR}]), region of screening, image style, and screening facility. Images were then regraded by a blinded, certified, secondary grader. Weighted kappa analysis was performed to evaluate agreement. RESULTS Fundus photographs from 230 patients (458 eyes) were received from practicing ophthalmologists (52.6%) and hospital-based grading centres (47.4%) from all Danish regions. Reported levels of DR by the primary graders were 66.8%, 12.2%, 13.1%, 1.3% and 5.5% for DR levels 0-4. The overall agreement between primary and secondary graders was 93% (κ = 0.83). Based on screening facility agreement was 96% (κ = 0.89) and 90% (κ = 0.76) for practicing ophthalmologists and hospital-based graders. CONCLUSION In this nationwide study, we observed a high overall inter-grader agreement and based on this, it is reasonable to assume that reported DR gradings in the screening programme in Denmark, accurately reflect the truth.
Collapse
Affiliation(s)
- Anne Suhr Thykjaer
- Department of Ophthalmology, Odense University Hospital, Odense, Denmark
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- Steno Diabetes Center Odense, Odense University Hospital, Odense, Denmark
| | - Jens Andresen
- Organization of Danish Practicing Ophthalmologists, Copenhagen, Denmark
| | - Nis Andersen
- Organization of Danish Practicing Ophthalmologists, Copenhagen, Denmark
| | - Toke Bek
- Department of Ophthalmology, Aarhus University Hospital, Aarhus, Denmark
| | - Steffen Heegaard
- Department of Ophthalmology, Rigshospitalet-Glostrup, Copenhagen, Denmark
| | - Javad Hajari
- Department of Ophthalmology, Rigshospitalet-Glostrup, Copenhagen, Denmark
| | | | - Sören Möller
- Research Unit OPEN, Department of Clinical Research, University of Southern Denmark, Odense, Denmark
| | - Frederik Nørregaard Pedersen
- Department of Ophthalmology, Odense University Hospital, Odense, Denmark
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
| | - Ryo Kawasaki
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- Department of Vision Informatics, University of Osaka, Osaka, Japan
| | - Kurt Højlund
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- Steno Diabetes Center Odense, Odense University Hospital, Odense, Denmark
| | - Katrine Hass Rubin
- Research Unit OPEN, Department of Clinical Research, University of Southern Denmark, Odense, Denmark
| | - Lonny Stokholm
- Research Unit OPEN, Department of Clinical Research, University of Southern Denmark, Odense, Denmark
| | - Tunde Peto
- Centre for Public Health, Queen's University Belfast, Belfast, UK
| | - Jakob Grauslund
- Department of Ophthalmology, Odense University Hospital, Odense, Denmark
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- Steno Diabetes Center Odense, Odense University Hospital, Odense, Denmark
| |
Collapse
|
3
|
Koseoglu ND, Grzybowski A, Liu TYA. Deep Learning Applications to Classification and Detection of Age-Related Macular Degeneration on Optical Coherence Tomography Imaging: A Review. Ophthalmol Ther 2023; 12:2347-2359. [PMID: 37493854 PMCID: PMC10441995 DOI: 10.1007/s40123-023-00775-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 07/14/2023] [Indexed: 07/27/2023] Open
Abstract
Age-related macular degeneration (AMD) is one of the leading causes of blindness in the elderly, more commonly in developed countries. Optical coherence tomography (OCT) is a non-invasive imaging device widely used for the diagnosis and management of AMD. Deep learning (DL) uses multilayered artificial neural networks (NN) for feature extraction, and is the cutting-edge technique for medical image analysis for diagnostic and prognostication purposes. Application of DL models to OCT image analysis has garnered significant interest in recent years. In this review, we aimed to summarize studies focusing on DL models used in classification and detection of AMD. Additionally, we provide a brief introduction to other DL applications in AMD, such as segmentation, prediction/prognostication, and models trained on multimodal imaging.
Collapse
Affiliation(s)
- Neslihan Dilruba Koseoglu
- Wilmer Eye Institute, Johns Hopkins University, 600 N. Wolfe St., Maumenee 726, Baltimore, MD, 21287, USA
| | - Andrzej Grzybowski
- Institute for Research in Ophthalmology, Foundation for Ophthalmology Development, Poznan, Poland
| | - T Y Alvin Liu
- Wilmer Eye Institute, Johns Hopkins University, 600 N. Wolfe St., Maumenee 726, Baltimore, MD, 21287, USA.
| |
Collapse
|
4
|
Triepels RJMA, Segers MHM, Rosen P, Nuijts RMMA, van den Biggelaar FJHM, Henry YP, Stenevi U, Tassignon MJ, Young D, Behndig A, Lundström M, Dickman MM. Development of machine learning models to predict posterior capsule rupture based on the EUREQUO registry. Acta Ophthalmol 2023. [PMID: 36789777 DOI: 10.1111/aos.15648] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 01/23/2023] [Accepted: 01/31/2023] [Indexed: 02/16/2023]
Abstract
PURPOSE To evaluate the performance of different probabilistic classifiers to predict posterior capsule rupture (PCR) prior to cataract surgery. METHODS Three probabilistic classifiers were constructed to estimate the probability of PCR: a Bayesian network (BN), logistic regression (LR) model, and multi-layer perceptron (MLP) network. The classifiers were trained on a sample of 2 853 376 surgeries reported to the European Registry of Quality Outcomes for Cataract and Refractive Surgery (EUREQUO) between 2008 and 2018. The performance of the classifiers was evaluated based on the area under the precision-recall curve (AUPRC) and compared to existing scoring models in the literature. Furthermore, direct risk factors for PCR were identified by analysing the independence structure of the BN. RESULTS The MLP network predicted PCR overall the best (AUPRC 13.1 ± 0.41%), followed by the BN (AUPRC 8.05 ± 0.39%) and the LR model (AUPRC 7.31 ± 0.15%). Direct risk factors for PCR include preoperative best-corrected visual acuity (BCVA), year of surgery, operation type, anaesthesia, target refraction, other ocular comorbidities, white cataract, and corneal opacities. CONCLUSIONS Our results suggest that the MLP network performs better than existing scoring models in the literature, despite a relatively low precision at high recall. Consequently, implementing the MLP network in clinical practice can potentially decrease the PCR rate.
Collapse
Affiliation(s)
- Ron J M A Triepels
- Department of Data Analytics and Digitalisation, Maastricht University, Maastricht, the Netherlands
| | - Maartje H M Segers
- University Eye Clinic, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Paul Rosen
- Department of Ophthalmology, Oxford Eye Hospital, Oxford, UK
| | - Rudy M M A Nuijts
- University Eye Clinic, Maastricht University Medical Center+, Maastricht, the Netherlands
| | | | - Ype P Henry
- Department of Ophthalmology, Amsterdam UMC, Amsterdam, the Netherlands
| | - Ulf Stenevi
- Department of Ophthalmology, Sahlgrenska University Hospital, Göteborg, Sweden
| | | | - David Young
- Department of Mathematics and Statistics, University of Strathclyde, Glasgow, UK
| | - Anders Behndig
- Department of Clinical Sciences, Ophthalmology, Umeå University, Umeå, Sweden
| | - Mats Lundström
- Department of Clinical Sciences, Ophthalmology, Lund University, Lund, Sweden
| | - Mor M Dickman
- University Eye Clinic, Maastricht University Medical Center+, Maastricht, the Netherlands
| |
Collapse
|
5
|
Potapenko I, Thiesson B, Kristensen M, Hajari JN, Ilginis T, Fuchs J, Hamann S, la Cour M. Automated artificial intelligence-based system for clinical follow-up of patients with age-related macular degeneration. Acta Ophthalmol 2022; 100:927-936. [PMID: 35322564 PMCID: PMC9790353 DOI: 10.1111/aos.15133] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2021] [Revised: 02/05/2022] [Accepted: 03/12/2022] [Indexed: 12/30/2022]
Abstract
PURPOSE In this study, we investigate the potential of a novel artificial intelligence-based system for autonomous follow-up of patients treated for neovascular age-related macular degeneration (AMD). METHODS A temporal deep learning model was trained on a data set of 84 489 optical coherence tomography scans from AMD patients to recognize disease activity, and its performance was compared with a published non-temporal model trained on the same data (Acta Ophthalmol, 2021). An autonomous follow-up system was created by augmenting the AI model with deterministic logic to suggest treatment according to the observe-and-plan regimen. To validate the AI-based system, a data set comprising clinical decisions and imaging data from 200 follow-up consultations was collected prospectively. In each case, both the autonomous AI decision and original clinical decision were compared with an expert panel consensus. RESULTS The temporal AI model proved superior at detecting disease activity compared with the model without temporal input (area under the curve 0.900 (95% CI 0.894-0.906) and 0.857 (95% CI 0.846-0.867) respectively). The AI-based follow-up system could make an autonomous decision in 73% of the cases, 91.8% of which were in agreement with expert consensus. This was on par with the 87.7% agreement rate between decisions made in the clinic and expert consensus (p = 0.33). CONCLUSIONS The proposed autonomous follow-up system was shown to be safe and compliant with expert consensus on par with clinical practice. The system could in the future ease the pressure on public ophthalmology services from an increasing number of AMD patients.
Collapse
Affiliation(s)
- Ivan Potapenko
- Department of OphthalmologyRigshospitaletCopenhagenDenmark,Faculty of Health and Medical SciencesUniversity of CopenhagenCopenhagenDenmark
| | - Bo Thiesson
- Enversion A/SAarhusDenmark,Department of EngineeringAarhus UniversityAarhusDenmark
| | | | | | - Tomas Ilginis
- Department of OphthalmologyRigshospitaletCopenhagenDenmark
| | - Josefine Fuchs
- Department of OphthalmologyRigshospitaletCopenhagenDenmark
| | - Steffen Hamann
- Department of OphthalmologyRigshospitaletCopenhagenDenmark,Faculty of Health and Medical SciencesUniversity of CopenhagenCopenhagenDenmark
| | - Morten la Cour
- Department of OphthalmologyRigshospitaletCopenhagenDenmark,Faculty of Health and Medical SciencesUniversity of CopenhagenCopenhagenDenmark
| |
Collapse
|
6
|
Alexopoulos P, Madu C, Wollstein G, Schuman JS. The Development and Clinical Application of Innovative Optical Ophthalmic Imaging Techniques. Front Med (Lausanne) 2022; 9:891369. [PMID: 35847772 PMCID: PMC9279625 DOI: 10.3389/fmed.2022.891369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2022] [Accepted: 05/23/2022] [Indexed: 11/22/2022] Open
Abstract
The field of ophthalmic imaging has grown substantially over the last years. Massive improvements in image processing and computer hardware have allowed the emergence of multiple imaging techniques of the eye that can transform patient care. The purpose of this review is to describe the most recent advances in eye imaging and explain how new technologies and imaging methods can be utilized in a clinical setting. The introduction of optical coherence tomography (OCT) was a revolution in eye imaging and has since become the standard of care for a plethora of conditions. Its most recent iterations, OCT angiography, and visible light OCT, as well as imaging modalities, such as fluorescent lifetime imaging ophthalmoscopy, would allow a more thorough evaluation of patients and provide additional information on disease processes. Toward that goal, the application of adaptive optics (AO) and full-field scanning to a variety of eye imaging techniques has further allowed the histologic study of single cells in the retina and anterior segment. Toward the goal of remote eye care and more accessible eye imaging, methods such as handheld OCT devices and imaging through smartphones, have emerged. Finally, incorporating artificial intelligence (AI) in eye images has the potential to become a new milestone for eye imaging while also contributing in social aspects of eye care.
Collapse
Affiliation(s)
- Palaiologos Alexopoulos
- Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, NY, United States
| | - Chisom Madu
- Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, NY, United States
| | - Gadi Wollstein
- Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, NY, United States
- Department of Biomedical Engineering, NYU Tandon School of Engineering, Brooklyn, NY, United States
- Center for Neural Science, College of Arts & Science, New York University, New York, NY, United States
| | - Joel S. Schuman
- Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, NY, United States
- Department of Biomedical Engineering, NYU Tandon School of Engineering, Brooklyn, NY, United States
- Center for Neural Science, College of Arts & Science, New York University, New York, NY, United States
- Department of Electrical and Computer Engineering, NYU Tandon School of Engineering, Brooklyn, NY, United States
| |
Collapse
|