1
|
Chen W, Miao C, Zhang Z, Fung CSH, Wang R, Chen Y, Qian Y, Cheng L, Yip KY, Tsui SKW, Cao Q. Commonly used software tools produce conflicting and overly-optimistic AUPRC values. Genome Biol 2024; 25:118. [PMID: 38741205 PMCID: PMC11089773 DOI: 10.1186/s13059-024-03266-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Accepted: 04/30/2024] [Indexed: 05/16/2024] Open
Abstract
The precision-recall curve (PRC) and the area under the precision-recall curve (AUPRC) are useful for quantifying classification performance. They are commonly used in situations with imbalanced classes, such as cancer diagnosis and cell type annotation. We evaluate 10 popular tools for plotting PRC and computing AUPRC, which were collectively used in more than 3000 published studies. We find the AUPRC values computed by the tools rank classifiers differently and some tools produce overly-optimistic results.
Collapse
Affiliation(s)
- Wenyu Chen
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Chen Miao
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Zhenghao Zhang
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Cathy Sin-Hang Fung
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Ran Wang
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Yizhen Chen
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Yan Qian
- The First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Lixin Cheng
- Shenzhen People's Hospital, First Affiliated Hospital of Southern University of Science and Technology, Second Clinical Medical College of Jinan University, Shenzhen, China
| | - Kevin Y Yip
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China.
- Sanford Burnham Prebys Medical Discovery Institute, La Jolla, CA, USA.
| | - Stephen Kwok-Wing Tsui
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China.
- Hong Kong Bioinformatics Centre, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China.
| | - Qin Cao
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China.
- Hong Kong Bioinformatics Centre, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China.
- Shenzhen Research Institute, The Chinese University of Hong Kong, Shenzhen, China.
| |
Collapse
|
2
|
Chen W, Miao C, Zhang Z, Fung CSH, Wang R, Chen Y, Qian Y, Cheng L, Yip KY, Tsui SKW, Cao Q. Commonly used software tools produce conflicting and overly-optimistic AUPRC values. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.02.578654. [PMID: 38370825 PMCID: PMC10871236 DOI: 10.1101/2024.02.02.578654] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
The precision-recall curve (PRC) and the area under it (AUPRC) are useful for quantifying classification performance. They are commonly used in situations with imbalanced classes, such as cancer diagnosis and cell type annotation. We evaluated 10 popular tools for plotting PRC and computing AUPRC, which were collectively used in >3,000 published studies. We found the AUPRC values computed by the tools rank classifiers differently and some tools produce overly-optimistic results.
Collapse
Affiliation(s)
- Wenyu Chen
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Chen Miao
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Zhenghao Zhang
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Cathy Sin-Hang Fung
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Ran Wang
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Yizhen Chen
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Yan Qian
- The First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Lixin Cheng
- Shenzhen People’s Hospital, First Affiliated Hospital of Southern University of Science and Technology, Second Clinical Medicine College of Jinan University, Shenzhen, China
| | - Kevin Y. Yip
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
- Sanford Burnham Prebys Medical Discovery Institute, La Jolla, CA, USA
| | - Stephen Kwok-Wing Tsui
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
- Hong Kong Bioinformatics Centre, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
| | - Qin Cao
- School of Biomedical Sciences, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
- Hong Kong Bioinformatics Centre, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China
- Shenzhen Research Institute, The Chinese University of Hong Kong, Shenzhen, China
| |
Collapse
|
3
|
Usman D, Abubakar MB, Ibrahim KG, Imam MU. Iron chelation and supplementation: A comparison in the management of inflammatory bowel disease using drosophila. Life Sci 2024; 336:122328. [PMID: 38061132 DOI: 10.1016/j.lfs.2023.122328] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2023] [Revised: 11/20/2023] [Accepted: 11/30/2023] [Indexed: 12/24/2023]
Abstract
AIMS Inflammatory Bowel Disease (IBD) is associated with systemic iron deficiency and has been managed with iron supplements which cause adverse side effects. Conversely, some reports highlight iron depletion to ameliorate IBD. The underlying intestinal response and comparative benefit of iron depletion and supplementation in IBD is unknown. The aims of this work were to characterize and compare the effects of iron supplementation and iron depletion in IBD. MAIN METHODS IBD was induced in Drosophila melanogaster using 3 % dextran sodium sulfate (DSS) in diet for 7 days. Using this model, we investigated the impacts of acute iron depletion (using bathophenanthroline disulfonate, BPS) and supplementation (using ferrous sulphate, FS), before and after IBD induction, on gut iron homeostasis, cell death, gut permeability, inflammation, antioxidant defence, antimicrobial response and several fly phenotypes. KEY FINDINGS DSS decreased fly mass (p < 0.001), increased gut permeability (p < 0.001) and shortened lifespan (p = 0.035) compared to control. The DSS-fed flies also showed significantly elevated lipid peroxidation (p < 0.001), and the upregulated expression of apoptotic marker- drice (p < 0.001), tight junction protein - bbg (p < 0.001), antimicrobial peptide - dpta (p = 0.002) and proinflammatory cytokine - upd2 (p < 0.001). BPS significantly (p < 0.05) increased fly mass and lifespan, decreased gut permeability, decreased lipid peroxidation and decreased levels of drice, bbg, dpta and upd2 in IBD flies. This iron chelation (using BPS) showed better protection from DSS-induced IBD than iron supplementation (using FS). Preventive and curative interventions, by BPS or FS, also differed in outcomes. SIGNIFICANCE This may inform precise management strategies aimed at tackling IBD and its recurrence.
Collapse
Affiliation(s)
- Dawoud Usman
- Centre for Advanced Medical Research and Training, Usmanu Danfodiyo University Sokoto, Nigeria; Department of Physiology, College of Health Sciences, Usmanu Danfodiyo University Sokoto, Nigeria
| | - Murtala Bello Abubakar
- Centre for Advanced Medical Research and Training, Usmanu Danfodiyo University Sokoto, Nigeria; Department of Physiology, College of Health Sciences, Usmanu Danfodiyo University Sokoto, Nigeria; Department of Physiology, College of Medicine and Health Sciences, Baze University, Abuja, Nigeria
| | - Kasimu Ghandi Ibrahim
- Centre for Advanced Medical Research and Training, Usmanu Danfodiyo University Sokoto, Nigeria; Department of Physiology, College of Health Sciences, Usmanu Danfodiyo University Sokoto, Nigeria; Department of Basic Medical and Dental Sciences, Faculty of Dentistry, Zarqa University, P. O. Box 2000, Zarqa 13110, Jordan; School of Physiology, Faculty of Health Sciences, University of the Witwatersrand, 7 York Road, Parktown 2193, Johannesburg, South Africa
| | - Mustapha Umar Imam
- Centre for Advanced Medical Research and Training, Usmanu Danfodiyo University Sokoto, Nigeria; Department of Medical Biochemistry, College of Health Sciences, Usmanu Danfodiyo University Sokoto, Nigeria.
| |
Collapse
|