1
|
Saeed SU, Ramalhinho J, Pinnock M, Shen Z, Fu Y, Montaña-Brown N, Bonmati E, Barratt DC, Pereira SP, Davidson B, Clarkson MJ, Hu Y. Active learning using adaptable task-based prioritisation. Med Image Anal 2024; 95:103181. [PMID: 38640779 DOI: 10.1016/j.media.2024.103181] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 04/03/2024] [Accepted: 04/12/2024] [Indexed: 04/21/2024]
Abstract
Supervised machine learning-based medical image computing applications necessitate expert label curation, while unlabelled image data might be relatively abundant. Active learning methods aim to prioritise a subset of available image data for expert annotation, for label-efficient model training. We develop a controller neural network that measures priority of images in a sequence of batches, as in batch-mode active learning, for multi-class segmentation tasks. The controller is optimised by rewarding positive task-specific performance gain, within a Markov decision process (MDP) environment that also optimises the task predictor. In this work, the task predictor is a segmentation network. A meta-reinforcement learning algorithm is proposed with multiple MDPs, such that the pre-trained controller can be adapted to a new MDP that contains data from different institutes and/or requires segmentation of different organs or structures within the abdomen. We present experimental results using multiple CT datasets from more than one thousand patients, with segmentation tasks of nine different abdominal organs, to demonstrate the efficacy of the learnt prioritisation controller function and its cross-institute and cross-organ adaptability. We show that the proposed adaptable prioritisation metric yields converging segmentation accuracy for a new kidney segmentation task, unseen in training, using between approximately 40% to 60% of labels otherwise required with other heuristic or random prioritisation metrics. For clinical datasets of limited size, the proposed adaptable prioritisation offers a performance improvement of 22.6% and 10.2% in Dice score, for tasks of kidney and liver vessel segmentation, respectively, compared to random prioritisation and alternative active sampling strategies.
Collapse
Affiliation(s)
- Shaheer U Saeed
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK.
| | - João Ramalhinho
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Mark Pinnock
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Ziyi Shen
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Yunguan Fu
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; InstaDeep, London, UK
| | - Nina Montaña-Brown
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Ester Bonmati
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; School of Computer Science and Engineering, University of Westminster, London, UK
| | - Dean C Barratt
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Stephen P Pereira
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; Institute for Liver and Digestive Health, University College London, London, UK
| | - Brian Davidson
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; Centre for Surgical Innovation, Organ Regeneration and Transplantation (CISORT), Division of Surgery & Interventional Science, University College London, London, UK
| | - Matthew J Clarkson
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Yipeng Hu
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| |
Collapse
|
2
|
Wang H, Jin Q, Li S, Liu S, Wang M, Song Z. A comprehensive survey on deep active learning in medical image analysis. Med Image Anal 2024; 95:103201. [PMID: 38776841 DOI: 10.1016/j.media.2024.103201] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Revised: 04/25/2024] [Accepted: 05/06/2024] [Indexed: 05/25/2024]
Abstract
Deep learning has achieved widespread success in medical image analysis, leading to an increasing demand for large-scale expert-annotated medical image datasets. Yet, the high cost of annotating medical images severely hampers the development of deep learning in this field. To reduce annotation costs, active learning aims to select the most informative samples for annotation and train high-performance models with as few labeled samples as possible. In this survey, we review the core methods of active learning, including the evaluation of informativeness and sampling strategy. For the first time, we provide a detailed summary of the integration of active learning with other label-efficient techniques, such as semi-supervised, self-supervised learning, and so on. We also summarize active learning works that are specifically tailored to medical image analysis. Additionally, we conduct a thorough comparative analysis of the performance of different AL methods in medical image analysis with experiments. In the end, we offer our perspectives on the future trends and challenges of active learning and its applications in medical image analysis. An accompanying paper list and code for the comparative analysis is available in https://github.com/LightersWang/Awesome-Active-Learning-for-Medical-Image-Analysis.
Collapse
Affiliation(s)
- Haoran Wang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China
| | - Qiuye Jin
- Computational Bioscience Research Center (CBRC), King Abdullah University of Science and Technology (KAUST), Thuwal 23955, Saudi Arabia
| | - Shiman Li
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China
| | - Siyu Liu
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China
| | - Manning Wang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China.
| | - Zhijian Song
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China.
| |
Collapse
|
3
|
Alahmari SS, Goldgof D, Hall LO, Mouton PR. A Review of Nuclei Detection and Segmentation on Microscopy Images Using Deep Learning With Applications to Unbiased Stereology Counting. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:7458-7477. [PMID: 36327184 DOI: 10.1109/tnnls.2022.3213407] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
The detection and segmentation of stained cells and nuclei are essential prerequisites for subsequent quantitative research for many diseases. Recently, deep learning has shown strong performance in many computer vision problems, including solutions for medical image analysis. Furthermore, accurate stereological quantification of microscopic structures in stained tissue sections plays a critical role in understanding human diseases and developing safe and effective treatments. In this article, we review the most recent deep learning approaches for cell (nuclei) detection and segmentation in cancer and Alzheimer's disease with an emphasis on deep learning approaches combined with unbiased stereology. Major challenges include accurate and reproducible cell detection and segmentation of microscopic images from stained sections. Finally, we discuss potential improvements and future trends in deep learning applied to cell detection and segmentation.
Collapse
|
4
|
Ding W, Jiang Y, Pang G, Liu Z, Wu Y, Li J, Wu F. AI model to detect contact relationship between maxillary sinus and posterior teeth. Heliyon 2024; 10:e31052. [PMID: 38799758 PMCID: PMC11126831 DOI: 10.1016/j.heliyon.2024.e31052] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Revised: 04/10/2024] [Accepted: 05/09/2024] [Indexed: 05/29/2024] Open
Abstract
Objectives To establish a novel deep learning networks (MSF-MPTnet) based on panoramic radiographs (PRs) for automatic assessment of relationship between maxillary sinus floor (MSF) and maxillary posterior teeth (MPT), and to compare accuracy of MSF-MPTnet, dentists and radiologists identifying contact relationship. Study design A total of 1035 PRs and 1035 Cone-beam computed tomographys (CBCT)images were collected from January 2018 to April 2022. The relationships were classified into class I and II by CBCT. Class I represents non-contact group, and class II represents contact group. 350 PRs were randomly selected as test dataset and accuracy of MSF-MPTnet, dentists, and radiologists was compared. Results The intraclass correlation coefficient of dentists was 0.460-0.690 and it was 0.453-0.664 for radiologists. Sensitivity and accuracy of MSF-MPTnet were 0.682-0.852and 0.890-0.951, indicating that the output performance of MSF-MPTnet was reliable. Accuracy of maxillary premolars and molars were 79.7%-90.3 %, 76.2%-89.2 % and 72.9%-88.3 % in MSF-MPTnet model, dentists and radiologists. Accuracy of class I relationship in the MSF-MPTnet model (67.7%-94.6 %) was higher than that of dentists (56.5%-84.6 %) in maxillary first premolars and right second premolar, and accuracy of class I relationship in the MSF-MPTnet model is also higher than radiologists (40.0%-78.1 %) in all teeth positions (p < 0.05). Conclusions MSF-MPTnet model could increase detecting accuracy of the relationship between MSF and MPT, minimize pseudo contact relationship and reduce frequency of CBCT use.
Collapse
Affiliation(s)
- Wanghui Ding
- Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Zhejiang Provincial Clinical Research Center for Oral Diseases, Key Laboratory of Oral Biomedical Research of Zhejiang Province, Cancer Center of Zhejiang University, Hangzhou, China
| | - Yindi Jiang
- Hangzhou Linping Traditional Chinese Medicine Hospital, China
| | - Gaozhi Pang
- College of Computer Science and Technology, Zhejiang University of Technology, China
| | - Ziang Liu
- Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Zhejiang Provincial Clinical Research Center for Oral Diseases, Key Laboratory of Oral Biomedical Research of Zhejiang Province, Cancer Center of Zhejiang University, Hangzhou, China
| | - Yuefan Wu
- Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Zhejiang Provincial Clinical Research Center for Oral Diseases, Key Laboratory of Oral Biomedical Research of Zhejiang Province, Cancer Center of Zhejiang University, Hangzhou, China
| | | | - Fuli Wu
- College of Computer Science and Technology, Zhejiang University of Technology, China
| |
Collapse
|
5
|
Huang S, Wang T, Xiong H, Wen B, Huan J, Dou D. Temporal Output Discrepancy for Loss Estimation-Based Active Learning. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:2109-2123. [PMID: 35853066 DOI: 10.1109/tnnls.2022.3186855] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
While deep learning succeeds in a wide range of tasks, it highly depends on the massive collection of annotated data which is expensive and time-consuming. To lower the cost of data annotation, active learning has been proposed to interactively query an oracle to annotate a small proportion of informative samples in an unlabeled dataset. Inspired by the fact that the samples with higher loss are usually more informative to the model than the samples with lower loss, in this article we present a novel deep active learning approach that queries the oracle for data annotation when the unlabeled sample is believed to incorporate high loss. The core of our approach is a measurement temporal output discrepancy (TOD) that estimates the sample loss by evaluating the discrepancy of outputs given by models at different optimization steps. Our theoretical investigation shows that TOD lower-bounds the accumulated sample loss thus it can be used to select informative unlabeled samples. On basis of TOD, we further develop an effective unlabeled data sampling strategy as well as an unsupervised learning criterion for active learning. Due to the simplicity of TOD, our methods are efficient, flexible, and task-agnostic. Extensive experimental results demonstrate that our approach achieves superior performances than the state-of-the-art active learning methods on image classification and semantic segmentation tasks. In addition, we show that TOD can be utilized to select the best model of potentially the highest testing accuracy from a pool of candidate models.
Collapse
|
6
|
Islam NU, Zhou Z, Gehlot S, Gotway MB, Liang J. Seeking an optimal approach for Computer-aided Diagnosis of Pulmonary Embolism. Med Image Anal 2024; 91:102988. [PMID: 37924750 PMCID: PMC11039560 DOI: 10.1016/j.media.2023.102988] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2022] [Revised: 09/28/2023] [Accepted: 09/29/2023] [Indexed: 11/06/2023]
Abstract
Pulmonary Embolism (PE) represents a thrombus ("blood clot"), usually originating from a lower extremity vein, that travels to the blood vessels in the lung, causing vascular obstruction and in some patients death. This disorder is commonly diagnosed using Computed Tomography Pulmonary Angiography (CTPA). Deep learning holds great promise for the Computer-aided Diagnosis (CAD) of PE. However, numerous deep learning methods, such as Convolutional Neural Networks (CNN) and Transformer-based models, exist for a given task, causing great confusion regarding the development of CAD systems for PE. To address this confusion, we present a comprehensive analysis of competing deep learning methods applicable to PE diagnosis based on four datasets. First, we use the RSNA PE dataset, which includes (weak) slice-level and exam-level labels, for PE classification and diagnosis, respectively. At the slice level, we compare CNNs with the Vision Transformer (ViT) and the Swin Transformer. We also investigate the impact of self-supervised versus (fully) supervised ImageNet pre-training, and transfer learning over training models from scratch. Additionally, at the exam level, we compare sequence model learning with our proposed transformer-based architecture, Embedding-based ViT (E-ViT). For the second and third datasets, we utilize the CAD-PE Challenge Dataset and Ferdowsi University of Mashad's PE Dataset, where we convert (strong) clot-level masks into slice-level annotations to evaluate the optimal CNN model for slice-level PE classification. Finally, we use our in-house PE-CAD dataset, which contains (strong) clot-level masks. Here, we investigate the impact of our vessel-oriented image representations and self-supervised pre-training on PE false positive reduction at the clot level across image dimensions (2D, 2.5D, and 3D). Our experiments show that (1) transfer learning boosts performance despite differences between photographic images and CTPA scans; (2) self-supervised pre-training can surpass (fully) supervised pre-training; (3) transformer-based models demonstrate comparable performance but slower convergence compared with CNNs for slice-level PE classification; (4) model trained on the RSNA PE dataset demonstrates promising performance when tested on unseen datasets for slice-level PE classification; (5) our E-ViT framework excels in handling variable numbers of slices and outperforms sequence model learning for exam-level diagnosis; and (6) vessel-oriented image representation and self-supervised pre-training both enhance performance for PE false positive reduction across image dimensions. Our optimal approach surpasses state-of-the-art results on the RSNA PE dataset, enhancing AUC by 0.62% (slice-level) and 2.22% (exam-level). On our in-house PE-CAD dataset, 3D vessel-oriented images improve performance from 80.07% to 91.35%, a remarkable 11% gain. Codes are available at GitHub.com/JLiangLab/CAD_PE.
Collapse
Affiliation(s)
- Nahid Ul Islam
- School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA
| | - Zongwei Zhou
- Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Shiv Gehlot
- Biomedical Informants Program, Arizona State University, Phoenix, AZ 85054, USA
| | | | - Jianming Liang
- Biomedical Informants Program, Arizona State University, Phoenix, AZ 85054, USA.
| |
Collapse
|
7
|
Lin R, Gholipour A, Thiran JP, Karimi D, Kebiri H, Cuadra MB. CROSS-AGE AND CROSS-SITE DOMAIN SHIFT IMPACTS ON DEEP LEARNING-BASED WHITE MATTER FIBER ESTIMATION IN NEWBORN AND BABY BRAINS. ARXIV 2023:arXiv:2312.14773v1. [PMID: 38196752 PMCID: PMC10775359] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Abstract
Deep learning models have shown great promise in estimating tissue microstructure from limited diffusion magnetic resonance imaging data. However, these models face domain shift challenges when test and train data are from different scanners and protocols, or when the models are applied to data with inherent variations such as the developing brains of infants and children scanned at various ages. Several techniques have been proposed to address some of these challenges, such as data harmonization or domain adaptation in the adult brain. However, those techniques remain unexplored for the estimation of fiber orientation distribution functions in the rapidly developing brains of infants. In this work, we extensively investigate the age effect and domain shift within and across two different cohorts of 201 newborns and 165 babies using the Method of Moments and fine-tuning strategies. Our results show that reduced variations in the microstructural development of babies in comparison to newborns directly impact the deep learning models' cross-age performance. We also demonstrate that a small number of target domain samples can significantly mitigate domain shift problems.
Collapse
Affiliation(s)
- Rizhong Lin
- Signal Processing Laboratory 5 (LTS5), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- Department of Radiology, Lausanne University Hospital (CHUV) and University of Lausanne (UNIL), Lausanne, Switzerland
- College of Electronic and Information Engineering, Tongji University, Shanghai, China
| | - Ali Gholipour
- Computational Radiology Laboratory, Department of Radiology, Boston Children's Hospital and Harvard Medical School, Boston, MA, USA
| | - Jean-Philippe Thiran
- Signal Processing Laboratory 5 (LTS5), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- Department of Radiology, Lausanne University Hospital (CHUV) and University of Lausanne (UNIL), Lausanne, Switzerland
- CIBM Center for Biomedical Imaging, Switzerland
| | - Davood Karimi
- Computational Radiology Laboratory, Department of Radiology, Boston Children's Hospital and Harvard Medical School, Boston, MA, USA
| | - Hamza Kebiri
- Department of Radiology, Lausanne University Hospital (CHUV) and University of Lausanne (UNIL), Lausanne, Switzerland
- CIBM Center for Biomedical Imaging, Switzerland
| | - Meritxell Bach Cuadra
- CIBM Center for Biomedical Imaging, Switzerland
- Department of Radiology, Lausanne University Hospital (CHUV) and University of Lausanne (UNIL), Lausanne, Switzerland
| |
Collapse
|
8
|
Zhou Z, Luo H, Pang J, Ding X, Gotway M, Liang J. Learning Anatomically Consistent Embedding for Chest Radiography. BMVC : PROCEEDINGS OF THE BRITISH MACHINE VISION CONFERENCE. BRITISH MACHINE VISION CONFERENCE 2023; 2023:617. [PMID: 38813080 PMCID: PMC11135486] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 05/31/2024]
Abstract
Self-supervised learning (SSL) approaches have recently shown substantial success in learning visual representations from unannotated images. Compared with photographic images, medical images acquired with the same imaging protocol exhibit high consistency in anatomy. To exploit this anatomical consistency, this paper introduces a novel SSL approach, called PEAC (patch embedding of anatomical consistency), for medical image analysis. Specifically, in this paper, we propose to learn global and local consistencies via stable grid-based matching, transfer pre-trained PEAC models to diverse downstream tasks, and extensively demonstrate that (1) PEAC achieves significantly better performance than the existing state-of-the-art fully/self-supervised methods, and (2) PEAC captures the anatomical structure consistency across views of the same patient and across patients of different genders, weights, and healthy statuses, which enhances the interpretability of our method for medical image analysis. All code and pretrained models are available at GitHub.com/JLiangLab/PEAC.
Collapse
Affiliation(s)
- Ziyu Zhou
- Shanghai Jiao Tong University, China
- Arizona State University, USA
| | | | | | | | | | | |
Collapse
|
9
|
Yu FH, Miao SM, Li CY, Hang J, Deng J, Ye XH, Liu Y. Pretreatment ultrasound-based deep learning radiomics model for the early prediction of pathologic response to neoadjuvant chemotherapy in breast cancer. Eur Radiol 2023; 33:5634-5644. [PMID: 36976336 DOI: 10.1007/s00330-023-09555-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Revised: 02/09/2023] [Accepted: 02/19/2023] [Indexed: 03/29/2023]
Abstract
OBJECTIVES To investigate the predictive performance of the deep learning radiomics (DLR) model integrating pretreatment ultrasound imaging features and clinical characteristics for evaluating therapeutic response after neoadjuvant chemotherapy (NAC) in patients with breast cancer. METHODS A total of 603 patients who underwent NAC were retrospectively included between January 2018 and June 2021 from three different institutions. Four different deep convolutional neural networks (DCNNs) were trained by pretreatment ultrasound images using annotated training dataset (n = 420) and validated in a testing cohort (n = 183). Comparing the predictive performance of these models, the best one was selected for image-only model structure. Furthermore, the integrated DLR model was constructed based on the image-only model combined with independent clinical-pathologic variables. Areas under the curve (AUCs) of these models and two radiologists were compared by using the DeLong method. RESULTS As the optimal basic model, Resnet50 achieved an AUC and accuracy of 0.879 and 82.5% in the validation set. The integrated DLR model, yielding the highest classification performance in predicting response to NAC (AUC 0.962 and 0.939 in the training and validation cohort), outperformed the image-only model and the clinical model and also performed better than two radiologists' prediction (all p < 0.05). In addition, predictive efficacy of the radiologists was improved under the assistance of the DLR model significantly. CONCLUSION The pretreatment US-based DLR model could hold promise as a clinical guidance for predicting NAC response of patients with breast cancer, thereby providing benefit of timely treatment strategy adjustment to potential poor NAC responders. KEY POINTS • Multicenter retrospective study showed that deep learning radiomics (DLR) model based on pretreatment ultrasound image and clinical parameter achieved satisfactory prediction of tumor response to neoadjuvant chemotherapy (NAC) in breast cancer. • The integrated DLR model could become an effective tool to guide clinicians in identifying potential poor pathological responders before chemotherapy. • The predictive efficacy of the radiologists was improved under the assistance of the DLR model.
Collapse
Affiliation(s)
- Fei-Hong Yu
- Department of Ultrasound, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
| | - Shu-Mei Miao
- Department of Information, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
- Department of Medical Informatics, School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, China
| | - Cui-Ying Li
- Department of Ultrasound, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
| | - Jing Hang
- Department of Ultrasound, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
| | - Jing Deng
- Department of Ultrasound, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
| | - Xin-Hua Ye
- Department of Ultrasound, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
| | - Yun Liu
- Department of Information, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China.
- Department of Medical Informatics, School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, China.
| |
Collapse
|
10
|
Kim JE, Choi YH, Lee YC, Seong G, Song JH, Kim TJ, Kim ER, Hong SN, Chang DK, Kim YH, Shin SY. Deep learning model for distinguishing Mayo endoscopic subscore 0 and 1 in patients with ulcerative colitis. Sci Rep 2023; 13:11351. [PMID: 37443370 PMCID: PMC10344868 DOI: 10.1038/s41598-023-38206-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2023] [Accepted: 07/05/2023] [Indexed: 07/15/2023] Open
Abstract
The aim of this study was to address the issue of differentiating between Mayo endoscopic subscore (MES) 0 and MES 1 using a deep learning model. A dataset of 492 ulcerative colitis (UC) patients who demonstrated MES improvement between January 2018 and December 2019 at Samsung Medical Center was utilized. Specifically, two representative images of the colon and rectum were selected from each patient, resulting in a total of 984 images for analysis. The deep learning model utilized in this study consisted of a convolutional neural network (CNN)-based encoder, with two auxiliary classifiers for the colon and rectum, as well as a final MES classifier that combined image features from both inputs. In the internal test, the model achieved an F1-score of 0.92, surpassing the performance of seven novice classifiers by an average margin of 0.11, and outperforming their consensus by 0.02. The area under the receiver operating characteristic curve (AUROC) was calculated to be 0.97 when considering MES 1 as positive, with an area under the precision-recall curve (AUPRC) of 0.98. In the external test using the Hyperkvasir dataset, the model achieved an F1-score of 0.89, AUROC of 0.86, and AUPRC of 0.97. The results demonstrate that the proposed CNN-based model, which integrates image features from both the colon and rectum, exhibits superior performance in accurately discriminating between MES 0 and MES 1 in patients with UC.
Collapse
Affiliation(s)
- Ji Eun Kim
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea
| | - Yoon Ho Choi
- Department of Artificial Intelligence and Informatics Research, Mayo Clinic, Jacksonville, FL, USA
- Department of Digital Health, Samsung Advanced Institute for Health Sciences and Technology, Sungkyunkwan University, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea
| | - Yeong Chan Lee
- Research Institute for Future Medicine, Samsung Medical Center, Seoul, South Korea
| | - Gyeol Seong
- Department of Medicine, Nowon Eulji Medical Center, Eulji University, Seoul, South Korea
| | - Joo Hye Song
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea
| | - Tae Jun Kim
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea
| | - Eun Ran Kim
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea
| | - Sung Noh Hong
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea
| | - Dong Kyung Chang
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea
| | - Young-Ho Kim
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea.
| | - Soo-Yong Shin
- Department of Digital Health, Samsung Advanced Institute for Health Sciences and Technology, Sungkyunkwan University, 81 Irwon-Ro, Gangnam-gu, Seoul, 06351, South Korea.
| |
Collapse
|
11
|
Lou W, Li H, Li G, Han X, Wan X. Which Pixel to Annotate: A Label-Efficient Nuclei Segmentation Framework. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:947-958. [PMID: 36355729 DOI: 10.1109/tmi.2022.3221666] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Recently deep neural networks, which require a large amount of annotated samples, have been widely applied in nuclei instance segmentation of H&E stained pathology images. However, it is inefficient and unnecessary to label all pixels for a dataset of nuclei images which usually contain similar and redundant patterns. Although unsupervised and semi-supervised learning methods have been studied for nuclei segmentation, very few works have delved into the selective labeling of samples to reduce the workload of annotation. Thus, in this paper, we propose a novel full nuclei segmentation framework that chooses only a few image patches to be annotated, augments the training set from the selected samples, and achieves nuclei segmentation in a semi-supervised manner. In the proposed framework, we first develop a novel consistency-based patch selection method to determine which image patches are the most beneficial to the training. Then we introduce a conditional single-image GAN with a component-wise discriminator, to synthesize more training samples. Lastly, our proposed framework trains an existing segmentation model with the above augmented samples. The experimental results show that our proposed method could obtain the same-level performance as a fully-supervised baseline by annotating less than 5% pixels on some benchmarks.
Collapse
|
12
|
Zhang M, Xue M, Li S, Zou Y, Zhu Q. Fusion deep learning approach combining diffuse optical tomography and ultrasound for improving breast cancer classification. BIOMEDICAL OPTICS EXPRESS 2023; 14:1636-1646. [PMID: 37078047 PMCID: PMC10110311 DOI: 10.1364/boe.486292] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/23/2023] [Revised: 02/25/2023] [Accepted: 03/04/2023] [Indexed: 05/03/2023]
Abstract
Diffuse optical tomography (DOT) is a promising technique that provides functional information related to tumor angiogenesis. However, reconstructing the DOT function map of a breast lesion is an ill-posed and underdetermined inverse process. A co-registered ultrasound (US) system that provides structural information about the breast lesion can improve the localization and accuracy of DOT reconstruction. Additionally, the well-known US characteristics of benign and malignant breast lesions can further improve cancer diagnosis based on DOT alone. Inspired by a fusion model deep learning approach, we combined US features extracted by a modified VGG-11 network with images reconstructed from a DOT deep learning auto-encoder-based model to form a new neural network for breast cancer diagnosis. The combined neural network model was trained with simulation data and fine-tuned with clinical data: it achieved an AUC of 0.931 (95% CI: 0.919-0.943), superior to those achieved using US images alone (0.860) or DOT images alone (0.842).
Collapse
Affiliation(s)
- Menghao Zhang
- Electrical and System Engineering Department, Washington University in St. Louis, 1 Brooking Dr, St. Louis, MO 63130, USA
| | - Minghao Xue
- Biomedical Engineering Department, Washington University in St. Louis, 1 Brooking Dr, St. Louis, MO 63130, USA
| | - Shuying Li
- Biomedical Engineering Department, Washington University in St. Louis, 1 Brooking Dr, St. Louis, MO 63130, USA
| | - Yun Zou
- Biomedical Engineering Department, Washington University in St. Louis, 1 Brooking Dr, St. Louis, MO 63130, USA
| | - Quing Zhu
- Electrical and System Engineering Department, Washington University in St. Louis, 1 Brooking Dr, St. Louis, MO 63130, USA
- Biomedical Engineering Department, Washington University in St. Louis, 1 Brooking Dr, St. Louis, MO 63130, USA
| |
Collapse
|
13
|
Onal Ertugrul I, Ahn YA, Bilalpur M, Messinger DS, Speltz ML, Cohn JF. Infant AFAR: Automated facial action recognition in infants. Behav Res Methods 2023; 55:1024-1035. [PMID: 35538295 PMCID: PMC9646921 DOI: 10.3758/s13428-022-01863-y] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/13/2022] [Indexed: 11/08/2022]
Abstract
Automated detection of facial action units in infants is challenging. Infant faces have different proportions, less texture, fewer wrinkles and furrows, and unique facial actions relative to adults. For these and related reasons, action unit (AU) detectors that are trained on adult faces may generalize poorly to infant faces. To train and test AU detectors for infant faces, we trained convolutional neural networks (CNN) in adult video databases and fine-tuned these networks in two large, manually annotated, infant video databases that differ in context, head pose, illumination, video resolution, and infant age. AUs were those central to expression of positive and negative emotion. AU detectors trained in infants greatly outperformed ones trained previously in adults. Training AU detectors across infant databases afforded greater robustness to between-database differences than did training database specific AU detectors and outperformed previous state-of-the-art in infant AU detection. The resulting AU detection system, which we refer to as Infant AFAR (Automated Facial Action Recognition), is available to the research community for further testing and applications in infant emotion, social interaction, and related topics.
Collapse
|
14
|
Wen G, Shim V, Holdsworth SJ, Fernandez J, Qiao M, Kasabov N, Wang A. Machine Learning for Brain MRI Data Harmonisation: A Systematic Review. Bioengineering (Basel) 2023; 10:bioengineering10040397. [PMID: 37106584 PMCID: PMC10135601 DOI: 10.3390/bioengineering10040397] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Revised: 03/16/2023] [Accepted: 03/21/2023] [Indexed: 04/29/2023] Open
Abstract
BACKGROUND Magnetic Resonance Imaging (MRI) data collected from multiple centres can be heterogeneous due to factors such as the scanner used and the site location. To reduce this heterogeneity, the data needs to be harmonised. In recent years, machine learning (ML) has been used to solve different types of problems related to MRI data, showing great promise. OBJECTIVE This study explores how well various ML algorithms perform in harmonising MRI data, both implicitly and explicitly, by summarising the findings in relevant peer-reviewed articles. Furthermore, it provides guidelines for the use of current methods and identifies potential future research directions. METHOD This review covers articles published through PubMed, Web of Science, and IEEE databases through June 2022. Data from studies were analysed based on the criteria of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). Quality assessment questions were derived to assess the quality of the included publications. RESULTS a total of 41 articles published between 2015 and 2022 were identified and analysed. In the review, MRI data has been found to be harmonised either in an implicit (n = 21) or an explicit (n = 20) way. Three MRI modalities were identified: structural MRI (n = 28), diffusion MRI (n = 7) and functional MRI (n = 6). CONCLUSION Various ML techniques have been employed to harmonise different types of MRI data. There is currently a lack of consistent evaluation methods and metrics used across studies, and it is recommended that the issue be addressed in future studies. Harmonisation of MRI data using ML shows promises in improving performance for ML downstream tasks, while caution should be exercised when using ML-harmonised data for direct interpretation.
Collapse
Affiliation(s)
- Grace Wen
- Auckland Bioengineering Institute, University of Auckland, Auckland 1142, New Zealand
| | - Vickie Shim
- Auckland Bioengineering Institute, University of Auckland, Auckland 1142, New Zealand
- Centre for Brain Research, University of Auckland, Auckland 1142, New Zealand
| | - Samantha Jane Holdsworth
- Centre for Brain Research, University of Auckland, Auckland 1142, New Zealand
- Mātai Medical Research Institute, Tairāwhiti-Gisborne 4010, New Zealand
- Department of Anatomy & Medical Imaging, Faculty of Medical and Health Sciences, University of Auckland, Auckland 1142, New Zealand
| | - Justin Fernandez
- Auckland Bioengineering Institute, University of Auckland, Auckland 1142, New Zealand
| | - Miao Qiao
- Department of Computer Science, University of Auckland, Auckland 1142, New Zealand
| | - Nikola Kasabov
- Auckland Bioengineering Institute, University of Auckland, Auckland 1142, New Zealand
- Knowledge Engineering and Discovery Research Institute, Auckland University of Technology, Auckland 1010, New Zealand
- Intelligent Systems Research Centre, Ulster University, Londonderry BT52 1SA, UK
- Institute for Information and Communication Technologies, Bulgarian Academy of Sciences, 1113 Sofia, Bulgaria
| | - Alan Wang
- Auckland Bioengineering Institute, University of Auckland, Auckland 1142, New Zealand
- Centre for Brain Research, University of Auckland, Auckland 1142, New Zealand
- Department of Anatomy & Medical Imaging, Faculty of Medical and Health Sciences, University of Auckland, Auckland 1142, New Zealand
| |
Collapse
|
15
|
Basu A, Senapati P, Deb M, Rai R, Dhal KG. A survey on recent trends in deep learning for nucleus segmentation from histopathology images. EVOLVING SYSTEMS 2023; 15:1-46. [PMID: 38625364 PMCID: PMC9987406 DOI: 10.1007/s12530-023-09491-3] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2022] [Accepted: 02/13/2023] [Indexed: 03/08/2023]
Abstract
Nucleus segmentation is an imperative step in the qualitative study of imaging datasets, considered as an intricate task in histopathology image analysis. Segmenting a nucleus is an important part of diagnosing, staging, and grading cancer, but overlapping regions make it hard to separate and tell apart independent nuclei. Deep Learning is swiftly paving its way in the arena of nucleus segmentation, attracting quite a few researchers with its numerous published research articles indicating its efficacy in the field. This paper presents a systematic survey on nucleus segmentation using deep learning in the last five years (2017-2021), highlighting various segmentation models (U-Net, SCPP-Net, Sharp U-Net, and LiverNet) and exploring their similarities, strengths, datasets utilized, and unfolding research areas.
Collapse
Affiliation(s)
- Anusua Basu
- Department of Computer Science and Application, Midnapore College (Autonomous), Paschim Medinipur, Midnapore, West Bengal India
| | - Pradip Senapati
- Department of Computer Science and Application, Midnapore College (Autonomous), Paschim Medinipur, Midnapore, West Bengal India
| | - Mainak Deb
- Wipro Technologies, Pune, Maharashtra India
| | - Rebika Rai
- Department of Computer Applications, Sikkim University, Sikkim, India
| | - Krishna Gopal Dhal
- Department of Computer Science and Application, Midnapore College (Autonomous), Paschim Medinipur, Midnapore, West Bengal India
| |
Collapse
|
16
|
Yamada T, Massot-Campos M, Prugel-Bennett A, Pizarro O, Williams SB, Thornton B. Guiding Labelling Effort for Efficient Learning With Georeferenced Images. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:593-607. [PMID: 34982674 DOI: 10.1109/tpami.2021.3140060] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
We describe a novel semi-supervised learning method that reduces the labelling effort needed to train convolutional neural networks (CNNs) when processing georeferenced imagery. This allows deep learning CNNs to be trained on a per-dataset basis, which is useful in domains where there is limited learning transferability across datasets. The method identifies representative subsets of images from an unlabelled dataset based on the latent representation of a location guided autoencoder. We assess the method's sensitivities to design options using four different ground-truthed datasets of georeferenced environmental monitoring images, where these include various scenes in aerial and seafloor imagery. Efficiency gains are achieved for all the aerial and seafloor image datasets analysed in our experiments, demonstrating the benefit of the method across application domains. Compared to CNNs of the same architecture trained using conventional transfer and active learning, the method achieves equivalent accuracy with an order of magnitude fewer annotations, and 85 % of the accuracy of CNNs trained conventionally with approximately 10,000 human annotations using just 40 prioritised annotations. The biggest gains in efficiency are seen in datasets with unbalanced class distributions and rare classes that have a relatively small number of observations.
Collapse
|
17
|
Wang X, Liang B, Yang M, Li W. What makes the unsupervised monocular depth estimation (UMDE) model training better. Sci Rep 2022; 12:21999. [PMID: 36539595 PMCID: PMC9768171 DOI: 10.1038/s41598-022-26613-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2022] [Accepted: 12/16/2022] [Indexed: 12/24/2022] Open
Abstract
Current computer vision tasks based on deep learning require a huge amount of data with annotations for model training or testing, especially in some dense estimation tasks, such as optical flow segmentation and depth estimation. In practice, manual labeling for dense estimation tasks is very difficult or even impossible, and the scenes of the dataset are often restricted to a small range, which dramatically limits the development of the community. To overcome this deficiency, we propose a synthetic dataset generation method to obtain the expandable dataset without burdensome manual workforce. By this method, we construct a dataset called MineNavi containing video footages from first-perspective-view of the aircraft matched with accurate ground truth for depth estimation in aircraft navigation application. We also provide quantitative experiments to prove that pre-training via our MineNavi dataset can improve the performance of depth estimation model and speed up the convergence of the model on real scene data. Since the synthetic dataset has a similar effect to the real-world dataset in the training process of deep model, we finally conduct the experiments on MineNavi with unsupervised monocular depth estimation (UMDE) deep learning models to demonstrate the impact of various factors in our dataset such as lighting conditions and motion mode, aiming to explore what makes this kind of models training better.
Collapse
Affiliation(s)
- Xiangtong Wang
- grid.13291.380000 0001 0807 1581School of Aeronautics and Astronautics, Sichuan University, Chengdu, China
| | - Binbin Liang
- grid.13291.380000 0001 0807 1581School of Aeronautics and Astronautics, Sichuan University, Chengdu, China
| | - Menglong Yang
- grid.13291.380000 0001 0807 1581School of Aeronautics and Astronautics, Sichuan University, Chengdu, China
| | - Wei Li
- grid.13291.380000 0001 0807 1581School of Aeronautics and Astronautics, Sichuan University, Chengdu, China
| |
Collapse
|
18
|
Ko YS, Choi YM, Kim M, Park Y, Ashraf M, Quiñones Robles WR, Kim MJ, Jang J, Yun S, Hwang Y, Jang H, Yi MY. Improving quality control in the routine practice for histopathological interpretation of gastrointestinal endoscopic biopsies using artificial intelligence. PLoS One 2022; 17:e0278542. [PMID: 36520777 PMCID: PMC9754254 DOI: 10.1371/journal.pone.0278542] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2022] [Accepted: 11/18/2022] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Colorectal and gastric cancer are major causes of cancer-related deaths. In Korea, gastrointestinal (GI) endoscopic biopsy specimens account for a high percentage of histopathologic examinations. Lack of a sufficient pathologist workforce can cause an increase in human errors, threatening patient safety. Therefore, we developed a digital pathology total solution combining artificial intelligence (AI) classifier models and pathology laboratory information system for GI endoscopic biopsy specimens to establish a post-analytic daily fast quality control (QC) system, which was applied in clinical practice for a 3-month trial run by four pathologists. METHODS AND FINDINGS Our whole slide image (WSI) classification framework comprised patch-generator, patch-level classifier, and WSI-level classifier. The classifiers were both based on DenseNet (Dense Convolutional Network). In laboratory tests, the WSI classifier achieved accuracy rates of 95.8% and 96.0% in classifying histopathological WSIs of colorectal and gastric endoscopic biopsy specimens, respectively, into three classes (Negative for dysplasia, Dysplasia, and Malignant). Classification by pathologic diagnosis and AI prediction were compared and daily reviews were conducted, focusing on discordant cases for early detection of potential human errors by the pathologists, allowing immediate correction, before the pathology report error is conveyed to the patients. During the 3-month AI-assisted daily QC trial run period, approximately 7-10 times the number of slides compared to that in the conventional monthly QC (33 months) were reviewed by pathologists; nearly 100% of GI endoscopy biopsy slides were double-checked by the AI models. Further, approximately 17-30 times the number of potential human errors were detected within an average of 1.2 days. CONCLUSIONS The AI-assisted daily QC system that we developed and established demonstrated notable improvements in QC, in quantitative, qualitative, and time utility aspects. Ultimately, we developed an independent AI-assisted post-analytic daily fast QC system that was clinically applicable and influential, which could enhance patient safety.
Collapse
Affiliation(s)
- Young Sin Ko
- Pathology Center, Seegene Medical Foundation, Seoul, Republic of Korea
- * E-mail: (YSK); (MYY)
| | - Yoo Mi Choi
- Pathology Center, Seegene Medical Foundation, Seoul, Republic of Korea
| | - Mujin Kim
- Graduate School of Data Science, Department of Industrial & Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Youngjin Park
- Graduate School of Data Science, Department of Industrial & Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Murtaza Ashraf
- Graduate School of Data Science, Department of Industrial & Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Willmer Rafell Quiñones Robles
- Graduate School of Data Science, Department of Industrial & Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Min-Ju Kim
- Department of Pathology, Incheon Sejong Hospital, Incheon, Republic of Korea
| | - Jiwook Jang
- AI Research Team, Digital Innovation Sector, Seegene Medical Foundation, Seoul, Republic of Korea
| | - Seokju Yun
- AI Research Team, Digital Innovation Sector, Seegene Medical Foundation, Seoul, Republic of Korea
| | - Yuri Hwang
- AI Research Team, Digital Innovation Sector, Seegene Medical Foundation, Seoul, Republic of Korea
| | - Hani Jang
- AI Research Team, Digital Innovation Sector, Seegene Medical Foundation, Seoul, Republic of Korea
| | - Mun Yong Yi
- Graduate School of Data Science, Department of Industrial & Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
- * E-mail: (YSK); (MYY)
| |
Collapse
|
19
|
Khan AI, Kim MJ, Dutta P. Fine-tuning-based Transfer Learning for Characterization of Adeno-Associated Virus. JOURNAL OF SIGNAL PROCESSING SYSTEMS 2022; 94:1515-1529. [PMID: 36742147 PMCID: PMC9897492 DOI: 10.1007/s11265-022-01758-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 04/01/2022] [Indexed: 06/18/2023]
Abstract
Accurate and precise identification of adeno-associated virus (AAV) vectors play an important role in dose-dependent gene therapy. Although solid-state nanopore techniques can potentially be used to characterize AAV vectors by capturing ionic current, the existing data analysis techniques fall short of identifying them from their ionic current profiles. Recently introduced machine learning methods such as deep convolutional neural network (CNN), developed for image identification tasks, can be applied for such classification. However, with smaller data set for the problem in hand, it is not possible to train a deep neural network from scratch for accurate classification of AAV vectors. To circumvent this, we applied a pre-trained deep CNN (GoogleNet) model to capture the basic features from ionic current signals and subsequently used fine-tuning-based transfer learning to classify AAV vectors. The proposed method is very generic as it requires minimal preprocessing and does not require any handcrafted features. Our results indicate that fine-tuning-based transfer learning can achieve an average classification accuracy between 90 and 99% in three realizations with a very small standard deviation. Results also indicate that the classification accuracy depends on the applied electric field (across nanopore) and the time frame used for data segmentation. We also found that the fine-tuning of the deep network outperforms feature extraction-based classification for the resistive pulse dataset. To expand the usefulness of the fine-tuning-based transfer learning, we have tested two other pre-trained deep networks (ResNet50 and InceptionV3) for the classification of AAVs. Overall, the fine-tuning-based transfer learning from pre-trained deep networks is very effective for classification, though deep networks such as ResNet50 and InceptionV3 take significantly longer training time than GoogleNet.
Collapse
Affiliation(s)
- Aminul Islam Khan
- School of Mechanical and Materials Engineering, Washington State University, Pullman, WA, 99164, USA
| | - Min Jun Kim
- Department of Mechanical Engineering, Southern Methodist University, Dallas, TX, 75275, USA
| | - Prashanta Dutta
- School of Mechanical and Materials Engineering, Washington State University, Pullman, WA, 99164, USA
| |
Collapse
|
20
|
Xia X, Wang J, Liang S, Ye F, Tian MM, Hu W, Xu L. An attention base U-net for parotid tumor autosegmentation. Front Oncol 2022; 12:1028382. [PMID: 36505865 PMCID: PMC9730401 DOI: 10.3389/fonc.2022.1028382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 10/26/2022] [Indexed: 11/25/2022] Open
Abstract
A parotid neoplasm is an uncommon condition that only accounts for less than 3% of all head and neck cancers, and they make up less than 0.3% of all new cancers diagnosed annually. Due to their nonspecific imaging features and heterogeneous nature, accurate preoperative diagnosis remains a challenge. Automatic parotid tumor segmentation may help physicians evaluate these tumors. Two hundred eighty-five patients diagnosed with benign or malignant parotid tumors were enrolled in this study. Parotid and tumor tissues were segmented by 3 radiologists on T1-weighted (T1w), T2-weighted (T2w) and T1-weighted contrast-enhanced (T1wC) MR images. These images were randomly divided into two datasets, including a training dataset (90%) and an validation dataset (10%). A 10-fold cross-validation was performed to assess the performance. An attention base U-net for parotid tumor autosegmentation was created on the MRI T1w, T2 and T1wC images. The results were evaluated in a separate dataset, and the mean Dice similarity coefficient (DICE) for both parotids was 0.88. The mean DICE for left and right tumors was 0.85 and 0.86, respectively. These results indicate that the performance of this model corresponds with the radiologist's manual segmentation. In conclusion, an attention base U-net for parotid tumor autosegmentation may assist physicians to evaluate parotid gland tumors.
Collapse
Affiliation(s)
- Xianwu Xia
- The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang, China,Department of Oncology Intervention, The Affiliated Municipal Hospital of Taizhou University, Taizhou, China,Department of Radiation Oncology, Fudan University Shanghai Cancer Center, Shanghai, China,Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China
| | - Jiazhou Wang
- Department of Radiation Oncology, Fudan University Shanghai Cancer Center, Shanghai, China,Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China
| | - Sheng Liang
- Department of Oncology Intervention, The Affiliated Municipal Hospital of Taizhou University, Taizhou, China
| | - Fangfang Ye
- Department of Oncology Intervention, The Affiliated Municipal Hospital of Taizhou University, Taizhou, China
| | - Min-Ming Tian
- Department of Oncology Intervention, Jiangxi University of Traditional Chinese Medicine, Nanchang, Jiangxi, China
| | - Weigang Hu
- Department of Radiation Oncology, Fudan University Shanghai Cancer Center, Shanghai, China,Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China,*Correspondence: Weigang Hu, ; Leiming Xu,
| | - Leiming Xu
- The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang, China,*Correspondence: Weigang Hu, ; Leiming Xu,
| |
Collapse
|
21
|
Gou F, Liu J, Zhu J, Wu J. A Multimodal Auxiliary Classification System for Osteosarcoma Histopathological Images Based on Deep Active Learning. Healthcare (Basel) 2022; 10:2189. [PMID: 36360530 PMCID: PMC9690420 DOI: 10.3390/healthcare10112189] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Revised: 10/27/2022] [Accepted: 10/28/2022] [Indexed: 10/29/2023] Open
Abstract
Histopathological examination is an important criterion in the clinical diagnosis of osteosarcoma. With the improvement of hardware technology and computing power, pathological image analysis systems based on artificial intelligence have been widely used. However, classifying numerous intricate pathology images by hand is a tiresome task for pathologists. The lack of labeling data makes the system costly and difficult to build. This study constructs a classification assistance system (OHIcsA) based on active learning (AL) and a generative adversarial network (GAN). The system initially uses a small, labeled training set to train the classifier. Then, the most informative samples from the unlabeled images are selected for expert annotation. To retrain the network, the final chosen images are added to the initial labeled dataset. Experiments on real datasets show that our proposed method achieves high classification performance with an AUC value of 0.995 and an accuracy value of 0.989 using a small amount of labeled data. It reduces the cost of building a medical system. Clinical diagnosis can be aided by the system's findings, which can also increase the effectiveness and verifiable accuracy of doctors.
Collapse
Affiliation(s)
- Fangfang Gou
- School of Computer Science and Engineering, Central South University, Changsha 410083, China
| | - Jun Liu
- The Second People’s Hospital of Huaihua, Huaihua 418000, China
| | - Jun Zhu
- The First People’s Hospital of Huaihua, Huaihua 418000, China
- Collaborative Innovation Center for Medical Artificial Intelligence and Big Data Decision Making Assistance, Hunan University of Medicine, Huaihua 418000, China
| | - Jia Wu
- School of Computer Science and Engineering, Central South University, Changsha 410083, China
- Research Center for Artificial Intelligence, Monash University, Melbourne, VIC 3800, Australia
| |
Collapse
|
22
|
Tariq A, Tang S, Sakhi H, Celi LA, Newsome JM, Rubin DL, Trivedi H, Gichoy JW, Patel B, Banerjee I. Graph-based Fusion Modeling and Explanation for Disease Trajectory Prediction. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2022:2022.10.25.22281469. [PMID: 36324799 PMCID: PMC9628192 DOI: 10.1101/2022.10.25.22281469] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
Abstract
We propose a relational graph to incorporate clinical similarity between patients while building personalized clinical event predictors with a focus on hospitalized COVID-19 patients. Our graph formation process fuses heterogeneous data, i.e., chest X-rays as node features and non-imaging EHR for edge formation. While node represents a snap-shot in time for a single patient, weighted edge structure encodes complex clinical patterns among patients. While age and gender have been used in the past for patient graph formation, our method incorporates complex clinical history while avoiding manual feature selection. The model learns from the patient's own data as well as patterns among clinically-similar patients. Our visualization study investigates the effects of 'neighborhood' of a node on its predictiveness and showcases the model's tendency to focus on edge-connected patients with highly suggestive clinical features common with the node. The proposed model generalizes well by allowing edge formation process to adapt to an external cohort.
Collapse
Affiliation(s)
| | - Siyi Tang
- Department of Electrical Engineering, Stanford University
| | - Hifza Sakhi
- Philadelphia College of Osteopathic Medicine - Georgia Campus
| | | | | | | | - Hari Trivedi
- Department of Radiology and Imaging Sciences, Emory University, GA
| | | | | | | |
Collapse
|
23
|
A novel optimized tiny YOLOv3 algorithm for the identification of objects in the lawn environment. Sci Rep 2022; 12:15124. [PMID: 36068288 PMCID: PMC9448759 DOI: 10.1038/s41598-022-19519-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Accepted: 08/30/2022] [Indexed: 11/25/2022] Open
Abstract
Based on the problem of insufficient accuracy of the original tiny YOLOv3 algorithm for object detection in a lawn environment, an Optimized tiny YOLOv3 algorithm with less computation and higher accuracy is proposed. Three reasons affect the accuracy of the original tiny YOLOv3 algorithm for detecting objects in a lawn environment. First, the backbone of the original algorithm is composed of a stack of a single convolutional layer and a max-pooling layer, which results in insufficient ability to extract feature information of objects. An enhancement module is proposed to enhance the feature extraction capability of the shallow layers of the network. Second, the information of the shallow convolutional layers of the backbone is not fully used, which results in insufficient detection capability for small objects. Third, the deep part of the backbone uses a convolutional layer with an excessive number of channels, which results in a large amount of computation. A multi-resolution fusion module is proposed to enhance the information interaction capability of the deep and shallow layers of the network, and reduce the computation. To verify the accuracy of this Optimized tiny YOLOv3 algorithm, the algorithm was tested on the dataset containing trunk, spherical tree and person, and compared with the current research. The results show that the algorithm proposed in this paper improves the detection accuracy while reducing the calculation.
Collapse
|
24
|
Li Z, Zhu Q, Zhang L, Yang X, Li Z, Fu J. A deep learning-based self-adapting ensemble method for segmentation in gynecological brachytherapy. Radiat Oncol 2022; 17:152. [PMID: 36064571 PMCID: PMC9446699 DOI: 10.1186/s13014-022-02121-3] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2022] [Accepted: 08/29/2022] [Indexed: 11/10/2022] Open
Abstract
Purpose Fast and accurate outlining of the organs at risk (OARs) and high-risk clinical tumor volume (HRCTV) is especially important in high-dose-rate brachytherapy due to the highly time-intensive online treatment planning process and the high dose gradient around the HRCTV. This study aims to apply a self-configured ensemble method for fast and reproducible auto-segmentation of OARs and HRCTVs in gynecological cancer. Materials and methods We applied nnU-Net (no new U-Net), an automatically adapted deep convolutional neural network based on U-Net, to segment the bladder, rectum and HRCTV on CT images in gynecological cancer. In nnU-Net, three architectures, including 2D U-Net, 3D U-Net and 3D-Cascade U-Net, were trained and finally ensembled. 207 cases were randomly chosen for training, and 30 for testing. Quantitative evaluation used well-established image segmentation metrics, including dice similarity coefficient (DSC), 95% Hausdorff distance (HD95%), and average surface distance (ASD). Qualitative analysis of automated segmentation results was performed visually by two radiation oncologists. The dosimetric evaluation was performed by comparing the dose-volume parameters of both predicted segmentation and human contouring. Results nnU-Net obtained high qualitative and quantitative segmentation accuracy on the test dataset and performed better than previously reported methods in bladder and rectum segmentation. In quantitative evaluation, 3D-Cascade achieved the best performance in the bladder (DSC: 0.936 ± 0.051, HD95%: 3.503 ± 1.956, ASD: 0.944 ± 0.503), rectum (DSC: 0.831 ± 0.074, HD95%: 7.579 ± 5.857, ASD: 3.6 ± 3.485), and HRCTV (DSC: 0.836 ± 0.07, HD95%: 7.42 ± 5.023, ASD: 2.094 ± 1.311). According to the qualitative evaluation, over 76% of the test data set had no or minor visually detectable errors in segmentation. Conclusion This work showed nnU-Net’s superiority in segmenting OARs and HRCTV in gynecological brachytherapy cases in our center, among which 3D-Cascade shows the highest accuracy in segmentation across different applicators and patient anatomy. Supplementary Information The online version contains supplementary material available at 10.1186/s13014-022-02121-3.
Collapse
Affiliation(s)
- Zhen Li
- Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Xuhui District, Shanghai, China
| | - Qingyuan Zhu
- Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Xuhui District, Shanghai, China
| | - Lihua Zhang
- Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Xuhui District, Shanghai, China
| | - Xiaojing Yang
- Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Xuhui District, Shanghai, China
| | - Zhaobin Li
- Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Xuhui District, Shanghai, China.
| | - Jie Fu
- Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Xuhui District, Shanghai, China.
| |
Collapse
|
25
|
Automatic scoring of COVID-19 severity in X-ray imaging based on a novel deep learning workflow. Sci Rep 2022; 12:12791. [PMID: 35896761 PMCID: PMC9326426 DOI: 10.1038/s41598-022-15013-z] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2022] [Accepted: 06/16/2022] [Indexed: 11/30/2022] Open
Abstract
In this study, we propose a two-stage workflow used for the segmentation and scoring of lung diseases. The workflow inherits quantification, qualification, and visual assessment of lung diseases on X-ray images estimated by radiologists and clinicians. It requires the fulfillment of two core stages devoted to lung and disease segmentation as well as an additional post-processing stage devoted to scoring. The latter integrated block is utilized, mainly, for the estimation of segment scores and computes the overall severity score of a patient. The models of the proposed workflow were trained and tested on four publicly available X-ray datasets of COVID-19 patients and two X-ray datasets of patients with no pulmonary pathology. Based on a combined dataset consisting of 580 COVID-19 patients and 784 patients with no disorders, our best-performing algorithm is based on a combination of DeepLabV3 + , for lung segmentation, and MA-Net, for disease segmentation. The proposed algorithms’ mean absolute error (MAE) of 0.30 is significantly reduced in comparison to established COVID-19 algorithms; BS-net and COVID-Net-S, possessing MAEs of 2.52 and 1.83 respectively. Moreover, the proposed two-stage workflow was not only more accurate but also computationally efficient, it was approximately 11 times faster than the mentioned methods. In summary, we proposed an accurate, time-efficient, and versatile approach for segmentation and scoring of lung diseases illustrated for COVID-19 and with broader future applications for pneumonia, tuberculosis, pneumothorax, amongst others.
Collapse
|
26
|
Zhao Z, Lu W, Zeng Z, Xu K, Veeravalli B, Guan C. Self-supervised Assisted Active Learning for Skin Lesion Segmentation. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:5043-5046. [PMID: 36085746 DOI: 10.1109/embc48229.2022.9871734] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Label scarcity has been a long-standing issue for biomedical image segmentation, due to high annotation costs and professional requirements. Recently, active learning (AL) strategies strive to reduce annotation costs by querying a small portion of data for annotation, receiving much traction in the field of medical imaging. However, most of the existing AL methods have to initialize models with some randomly selected samples followed by active selection based on various criteria, such as uncertainty and diversity. Such random-start initialization methods inevitably introduce under-value redundant samples and unnecessary annotation costs. For the purpose of addressing the issue, we propose a novel self-supervised assisted active learning framework in the cold-start setting, in which the segmentation model is first warmed up with self-supervised learning (SSL), and then SSL features are used for sample selection via latent feature clustering without accessing labels. We assess our proposed methodology on skin lesions segmentation task. Extensive experiments demonstrate that our approach is capable of achieving promising performance with substantial improvements over existing baselines. Clinical Relevance- The proposed method can smartly select samples to annotate without requiring labels for model initialization, which can save annotation costs in clinical practice.
Collapse
|
27
|
Wang X, Xiang X, Zhang B, Liu X, Zheng J, Hu Q. Weakly Supervised Object Detection Based on Active Learning. Neural Process Lett 2022. [DOI: 10.1007/s11063-022-10855-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
28
|
Deep Active Learning Framework for Lymph Node Metastasis Prediction in Medical Support System. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:4601696. [PMID: 35592722 PMCID: PMC9113892 DOI: 10.1155/2022/4601696] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/21/2022] [Revised: 03/27/2022] [Accepted: 04/23/2022] [Indexed: 12/21/2022]
Abstract
Assessing the extent of cancer spread by histopathological analysis of sentinel axillary lymph nodes is an important part of breast cancer staging. With the maturity and prevalence of deep learning technology, building auxiliary medical systems can help to relieve the burden of pathologists and increase the diagnostic precision and accuracy during this process. However, such histopathological images have complex patterns that are difficult for ordinary people to understand and require professional medical practitioners to annotate. This increases the cost of constructing such medical systems. To reduce the cost of annotating and improve the performance of the model as much as possible, in other words, using as few labeled samples as possible to obtain a greater performance improvement, we propose a deep learning framework with a three-stage query strategy and novel model update strategy. The framework first trains an auto-encoder with all the samples to obtain a global representation in a low-dimensional space. In the query stage, the unlabeled samples are first selected according to uncertainty, and then, coreset-based methods are employed to reduce sample redundancy. Finally, distribution differences between labeled samples and unlabeled samples are evaluated and samples that can quickly eliminate the distribution differences are selected. This method achieves faster iterative efficiency than the uncertainty strategies, representative strategies, or hybrid strategies on the lymph node slice dataset and other commonly used datasets. It reaches the performance of training with all data, but only uses 50% of the labeled. During the model update process, we randomly freeze some weights and only train the task model on new labeled samples with a smaller learning rate. Compared with fine-tuning task model on new samples, large-scale performance degradation is avoided. Compared with the retraining strategy or the replay strategy, it reduces the training cost of updating the task model by 79.87% and 90.07%, respectively.
Collapse
|
29
|
Zhang H, Lian Q, Zhao J, Wang Y, Yang Y, Feng S. RatUNet: residual U-Net based on attention mechanism for image denoising. PeerJ Comput Sci 2022; 8:e970. [PMID: 35634105 PMCID: PMC9138094 DOI: 10.7717/peerj-cs.970] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Accepted: 04/11/2022] [Indexed: 06/15/2023]
Abstract
Deep convolutional neural networks (CNNs) have been very successful in image denoising. However, with the growth of the depth of plain networks, CNNs may result in performance degradation. The lack of network depth leads to the limited ability of the network to extract image features and difficults to fuse the shallow image features into the deep image information. In this work, we propose an improved deep convolutional U-Net framework (RatUNet) for image denoising. RatUNet improves Unet as follows: (1) RatUNet uses the residual blocks of ResNet to deepen the network depth, so as to avoid the network performance saturation. (2) RatUNet improves the down-sampling method, which is conducive to extracting image features. (3) RatUNet improves the up-sampling method, which is used to restore image details. (4) RatUNet improves the skip-connection method of the U-Net network, which is used to fuse the shallow feature information into the deep image details, and it is more conducive to restore the clean image. (5) In order to better process the edge information of the image, RatUNet uses depthwise and polarized self-attention mechanism to guide a CNN for image denoising. Extensive experiments show that our RatUNet is more efficient and has better performance than existing state-of-the-art denoising methods, especially in SSIM metrics, the denoising effect of the RatUNet achieves very high performance. Visualization results show that the denoised image by RatUNet is smoother and sharper than other methods.
Collapse
Affiliation(s)
- Huibin Zhang
- Institute of Information Science and Technology, Yanshan University, Qinhuang Dao, Hebei Province, China
- Computer Department, Xinzhou Teachers University, Xinzhou, Shanxi Province, China
| | - Qiusheng Lian
- Institute of Information Science and Technology, Yanshan University, Qinhuang Dao, Hebei Province, China
- Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qin Huangdao, Hebei Province, China
| | - Jianmin Zhao
- Institute of Information Science and Technology, Yanshan University, Qinhuang Dao, Hebei Province, China
- School of Information Engineering, Inner Mongolia University of Science and Technology, Baotou, Inner Mongolia Province, China
| | - Yining Wang
- Computer Department, Xinzhou Teachers University, Xinzhou, Shanxi Province, China
| | - Yuchi Yang
- Institute of Information Science and Technology, Yanshan University, Qinhuang Dao, Hebei Province, China
- Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qin Huangdao, Hebei Province, China
| | - Suqin Feng
- Computer Department, Xinzhou Teachers University, Xinzhou, Shanxi Province, China
| |
Collapse
|
30
|
Tavanapong W, Oh J, Riegler MA, Khaleel M, Mittal B, de Groen PC. Artificial Intelligence for Colonoscopy: Past, Present, and Future. IEEE J Biomed Health Inform 2022; 26:3950-3965. [PMID: 35316197 PMCID: PMC9478992 DOI: 10.1109/jbhi.2022.3160098] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
Abstract
During the past decades, many automated image analysis methods have been developed for colonoscopy. Real-time implementation of the most promising methods during colonoscopy has been tested in clinical trials, including several recent multi-center studies. All trials have shown results that may contribute to prevention of colorectal cancer. We summarize the past and present development of colonoscopy video analysis methods, focusing on two categories of artificial intelligence (AI) technologies used in clinical trials. These are (1) analysis and feedback for improving colonoscopy quality and (2) detection of abnormalities. Our survey includes methods that use traditional machine learning algorithms on carefully designed hand-crafted features as well as recent deep-learning methods. Lastly, we present the gap between current state-of-the-art technology and desirable clinical features and conclude with future directions of endoscopic AI technology development that will bridge the current gap.
Collapse
|
31
|
Mary Shyni H, Chitra E. A COMPARATIVE STUDY OF X-RAY AND CT IMAGES IN COVID-19 DETECTION USING IMAGE PROCESSING AND DEEP LEARNING TECHNIQUES. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE UPDATE 2022; 2:100054. [PMID: 35281724 PMCID: PMC8898857 DOI: 10.1016/j.cmpbup.2022.100054] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Abstract
The deadly coronavirus has not just devastated the lives of millions but has put the entire healthcare system under tremendous pressure. Early diagnosis of COVID-19 plays a significant role in isolating the positive cases and preventing the further spread of the disease. The medical images along with deep learning models provided faster and more accurate results in the detection of COVID-19. This article extensively reviews the recent deep learning techniques for COVID-19 diagnosis. The research articles discussed reveal that Convolutional Neural Network (CNN) is the most popular deep learning algorithm in detecting COVID-19 from medical images. An overview of the necessity of pre-processing the medical images, transfer learning and data augmentation techniques to deal with data scarcity problems, use of pre-trained models to save time and the role of medical images in the automatic detection of COVID-19 are summarized. This article also provides a sensible outlook for the young researchers to develop highly effective CNN models coupled with medical images in the early detection of the disease.
Collapse
Affiliation(s)
- H Mary Shyni
- Department of Electronics and Communication Engineering, SRM Institute of Science and Technology, Kattankulathur, Chennai, Tamil Nadu, India
| | - E Chitra
- Department of Electronics and Communication Engineering, SRM Institute of Science and Technology, Kattankulathur, Chennai, Tamil Nadu, India
| |
Collapse
|
32
|
Hao R, Liu L, Zhang J, Wang X, Liu J, Du X, He W, Liao J, Liu L, Mao Y. A Data-Efficient Framework for the Identification of Vaginitis Based on Deep Learning. JOURNAL OF HEALTHCARE ENGINEERING 2022; 2022:1929371. [PMID: 35265294 PMCID: PMC8898862 DOI: 10.1155/2022/1929371] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/29/2021] [Accepted: 02/09/2022] [Indexed: 11/18/2022]
Abstract
Vaginitis is a gynecological disease affecting the health of millions of women all over the world. The traditional diagnosis of vaginitis is based on manual microscopy, which is time-consuming and tedious. The deep learning method offers a fast and reliable solution for an automatic early diagnosis of vaginitis. However, deep neural networks require massive well-annotated data. Manual annotation of microscopic images is highly cost extensive because it not only is a time-consuming process but also needs highly trained people (doctors, pathologists, or technicians). Most existing active learning approaches are not applicable in microscopic images due to the nature of complex backgrounds and numerous formed elements. To address the problem of high cost of labeling microscopic images, we present a data-efficient framework for the identification of vaginitis based on transfer learning and active learning strategies. The proposed informative sample selection strategy selected the minimal training subset, and then the pretrained convolutional neural network (CNN) was fine-tuned on the selected subset. The experiment results show that the proposed pipeline can save 37.5% annotation cost while maintaining competitive performance. The proposed promising novel framework can significantly save the annotation cost and has the potential of extending widely to other microscopic imaging applications, such as blood microscopic image analysis.
Collapse
Affiliation(s)
- Ruqian Hao
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Lin Liu
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Jing Zhang
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Xiangzhou Wang
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Juanxiu Liu
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Xiaohui Du
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Wen He
- The Sixth People's Hospital of Chengdu, Chengdu 610051, China
| | - Jicheng Liao
- The Sixth People's Hospital of Chengdu, Chengdu 610051, China
| | - Lu Liu
- The Sixth People's Hospital of Chengdu, Chengdu 610051, China
| | - Yuanying Mao
- The Sixth People's Hospital of Chengdu, Chengdu 610051, China
| |
Collapse
|
33
|
Sutton RT, Zai Ane OR, Goebel R, Baumgart DC. Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy images. Sci Rep 2022; 12:2748. [PMID: 35177717 PMCID: PMC8854553 DOI: 10.1038/s41598-022-06726-2] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2021] [Accepted: 02/04/2022] [Indexed: 02/07/2023] Open
Abstract
Endoscopic evaluation to reliably grade disease activity, detect complications including cancer and verification of mucosal healing are paramount in the care of patients with ulcerative colitis (UC); but this evaluation is hampered by substantial intra- and interobserver variability. Recently, artificial intelligence methodologies have been proposed to facilitate more objective, reproducible endoscopic assessment. In a first step, we compared how well several deep learning convolutional neural network architectures (CNNs) applied to a diverse subset of 8000 labeled endoscopic still images derived from HyperKvasir, the largest multi-class image and video dataset from the gastrointestinal tract available today. The HyperKvasir dataset includes 110,079 images and 374 videos and could (1) accurately distinguish UC from non-UC pathologies, and (2) inform the Mayo score of endoscopic disease severity. We grouped 851 UC images labeled with a Mayo score of 0-3, into an inactive/mild (236) and moderate/severe (604) dichotomy. Weights were initialized with ImageNet, and Grid Search was used to identify the best hyperparameters using fivefold cross-validation. The best accuracy (87.50%) and Area Under the Curve (AUC) (0.90) was achieved using the DenseNet121 architecture, compared to 72.02% and 0.50 by predicting the majority class ('no skill' model). Finally, we used Gradient-weighted Class Activation Maps (Grad-CAM) to improve visual interpretation of the model and take an explainable artificial intelligence approach (XAI).
Collapse
Affiliation(s)
- Reed T Sutton
- Division of Gastroenterology, University of Alberta, 130 University Campus, Edmonton, AB, T6G 2X8, Canada
| | - Osmar R Zai Ane
- Department of Computing Science, University of Alberta, Edmonton, AB, Canada
- Alberta Machine Intelligence Institute, University of Alberta, Edmonton, AB, Canada
| | - Randolph Goebel
- Department of Computing Science, University of Alberta, Edmonton, AB, Canada
- Alberta Machine Intelligence Institute, University of Alberta, Edmonton, AB, Canada
| | - Daniel C Baumgart
- Division of Gastroenterology, University of Alberta, 130 University Campus, Edmonton, AB, T6G 2X8, Canada.
- Department of Computing Science, University of Alberta, Edmonton, AB, Canada.
| |
Collapse
|
34
|
Zhou Y, Ji R, Sun X, Su J, Meng D, Gao Y, Shen C. Plenty is Plague: Fine-Grained Learning for Visual Question Answering. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:697-709. [PMID: 31796387 DOI: 10.1109/tpami.2019.2956699] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Visual Question Answering (VQA) has attracted extensive research focus recently. Along with the ever-increasing data scale and model complexity, the enormous training cost has become an emerging challenge for VQA. In this article, we show such a massive training cost is indeed plague. In contrast, a fine-grained design of the learning paradigm can be extremely beneficial in terms of both training efficiency and model accuracy. In particular, we argue that there exist two essential and unexplored issues in the existing VQA training paradigm that randomly samples data in each epoch, namely, the "difficulty diversity" and the "label redundancy". Concretely, "difficulty diversity" refers to the varying difficulty levels of different question types, while "label redundancy" refers to the redundant and noisy labels contained in individual question type. To tackle these two issues, in this article we propose a fine-grained VQA learning paradigm with an actor-critic based learning agent, termed FG-A1C. Instead of using all training data from scratch, FG-A1C includes a learning agent that adaptively and intelligently schedules the most difficult question types in each training epoch. Subsequently, two curriculum learning based schemes are further designed to identify the most useful data to be learned within each inidividual question type. We conduct extensive experiments on the VQA2.0 and VQA-CP v2 datasets, which demonstrate the significant benefits of our approach. For instance, on VQA-CP v2, with less than 75 percent of the training data, our learning paradigms can help the model achieves better performance than using the whole dataset. Meanwhile, we also shows the effectivenesss of our method in guiding data labeling. Finally, the proposed paradigm can be seamlessly integrated with any cutting-edge VQA models, without modifying their structures.
Collapse
|
35
|
Peyvandi A, Majidi B, Peyvandi S, Patra JC, Moshiri B. Location-aware hazardous litter management for smart emergency governance in urban eco-cyber-physical systems. MULTIMEDIA TOOLS AND APPLICATIONS 2022; 81:22185-22214. [PMID: 35002472 PMCID: PMC8721641 DOI: 10.1007/s11042-021-11654-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/28/2020] [Revised: 09/21/2021] [Accepted: 09/28/2021] [Indexed: 06/14/2023]
Abstract
Smart city management is facing a new challenge from littered face masks during COVID-19 pandemic. Addressing the issues of detection and collection of this hazardous waste that is littered in public spaces and outside the controlled environments, usually associated with biomedical waste, is urgent for the safety of the communities around the world. Manual management of this waste is beyond the capabilities of governments worldwide as the geospatial scale of littering is very high and also because this contaminated litter is a health and safety issue for the waste collectors. In this paper, an autonomous biomedical waste management framework that uses edge surveillance and location intelligence for detection of the littered face masks and predictive modelling for emergency response to this problem is proposed. In this research a novel dataset of littered face masks in various conditions and environments is collected. Then, a new deep neural network architecture for rapid detection of discarded face masks on the video surveillance edge nodes is proposed. Furthermore, a location intelligence model for prediction of the areas with higher probability of hazardous litter in the smart city is presented. Experimental results show that the accuracy of the proposed model for detection of littered face masks in various environments is 96%, while the speed of processing is ten times faster than comparable models. The proposed framework can help authorities to plan for timely emergency response to scattering of hazardous material in residential environments.
Collapse
Affiliation(s)
- Amirhossein Peyvandi
- Department of Computer Engineering, Faculty of Engineering, Khatam University, Tehran, Iran
| | - Babak Majidi
- Department of Computer Engineering, Faculty of Engineering, Khatam University, Tehran, Iran
- Emergency and Rapid Response Simulation (ADERSIM) Artificial Intelligence Group, Faculty of Liberal Arts & Professional Studies, York University, Toronto, Canada
| | - Soodeh Peyvandi
- Business Intelligence, University of Applied Sciences Upper Austria, Steyr, Austria
| | - Jagdish C. Patra
- Faculty of Science, Engineering and Technology, Swinburne University of Technology, Melbourne, Australia
| | - Behzad Moshiri
- School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran
- Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada
| |
Collapse
|
36
|
Zabihollahy F, Viswanathan AN, Schmidt EJ, Morcos M, Lee J. Fully automated multiorgan segmentation of female pelvic magnetic resonance images with coarse-to-fine convolutional neural network. Med Phys 2021; 48:7028-7042. [PMID: 34609756 PMCID: PMC8597653 DOI: 10.1002/mp.15268] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2021] [Revised: 08/25/2021] [Accepted: 09/17/2021] [Indexed: 02/03/2023] Open
Abstract
PURPOSE Brachytherapy combined with external beam radiotherapy (EBRT) is the standard treatment for cervical cancer and has been shown to improve overall survival rates compared to EBRT only. Magnetic resonance (MR) imaging is used for radiotherapy (RT) planning and image guidance due to its excellent soft tissue image contrast. Rapid and accurate segmentation of organs at risk (OAR) is a crucial step in MR image-guided RT. In this paper, we propose a fully automated two-step convolutional neural network (CNN) approach to delineate multiple OARs from T2-weighted (T2W) MR images. METHODS We employ a coarse-to-fine segmentation strategy. The coarse segmentation step first identifies the approximate boundary of each organ of interest and crops the MR volume around the centroid of organ-specific region of interest (ROI). The cropped ROI volumes are then fed to organ-specific fine segmentation networks to produce detailed segmentation of each organ. A three-dimensional (3-D) U-Net is trained to perform the coarse segmentation. For the fine segmentation, a 3-D Dense U-Net is employed in which a modified 3-D dense block is incorporated into the 3-D U-Net-like network to acquire inter and intra-slice features and improve information flow while reducing computational complexity. Two sets of T2W MR images (221 cases for MR1 and 62 for MR2) were taken with slightly different imaging parameters and used for our network training and test. The network was first trained on MR1 which was a larger sample set. The trained model was then transferred to the MR2 domain via a fine-tuning approach. Active learning strategy was utilized for selecting the most valuable data from MR2 to be included in the adaptation via transfer learning. RESULTS The proposed method was tested on 20 MR1 and 32 MR2 test sets. Mean ± SD dice similarity coefficients are 0.93 ± 0.04, 0.87 ± 0.03, and 0.80 ± 0.10 on MR1 and 0.94 ± 0.05, 0.88 ± 0.04, and 0.80 ± 0.05 on MR2 for bladder, rectum, and sigmoid, respectively. Hausdorff distances (95th percentile) are 4.18 ± 0.52, 2.54 ± 0.41, and 5.03 ± 1.31 mm on MR1 and 2.89 ± 0.33, 2.24 ± 0.40, and 3.28 ± 1.08 mm on MR2, respectively. The performance of our method is superior to other state-of-the-art segmentation methods. CONCLUSIONS We proposed a two-step CNN approach for fully automated segmentation of female pelvic MR bladder, rectum, and sigmoid from T2W MR volume. Our experimental results demonstrate that the developed method is accurate, fast, and reproducible, and outperforms alternative state-of-the-art methods for OAR segmentation significantly (p < 0.05).
Collapse
Affiliation(s)
- Fatemeh Zabihollahy
- Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University, Baltimore, MD 21287, USA
| | - Akila N Viswanathan
- Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University, Baltimore, MD 21287, USA
| | - Ehud J Schmidt
- Division of Cardiology, Department of Medicine, Johns Hopkins University, Baltimore, MD 21287, USA
| | - Marc Morcos
- Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University, Baltimore, MD 21287, USA
| | - Junghoon Lee
- Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University, Baltimore, MD 21287, USA
| |
Collapse
|
37
|
Cai L, Xu X, Zhang L, Foo CS. Exploring Spatial Diversity for Region-Based Active Learning. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:8702-8712. [PMID: 34665728 DOI: 10.1109/tip.2021.3120041] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
State-of-the-art methods for semantic segmentation are based on deep neural networks trained on large-scale labeled datasets. Acquiring such datasets would incur large annotation costs, especially for dense pixel-level prediction tasks like semantic segmentation. We consider region-based active learning as a strategy to reduce annotation costs while maintaining high performance. In this setting, batches of informative image regions instead of entire images are selected for labeling. Importantly, we propose that enforcing local spatial diversity is beneficial for active learning in this case, and to incorporate spatial diversity along with the traditional active selection criterion, e.g., data sample uncertainty, in a unified optimization framework for region-based active learning. We apply this framework to the Cityscapes and PASCAL VOC datasets and demonstrate that the inclusion of spatial diversity effectively improves the performance of uncertainty-based and feature diversity-based active learning methods. Our framework achieves 95% performance of fully supervised methods with only 5 - 9% of the labeled pixels, outperforming all state-of-the-art region-based active learning methods for semantic segmentation.
Collapse
|
38
|
Webb JM, Adusei SA, Wang Y, Samreen N, Adler K, Meixner DD, Fazzio RT, Fatemi M, Alizad A. Comparing deep learning-based automatic segmentation of breast masses to expert interobserver variability in ultrasound imaging. Comput Biol Med 2021; 139:104966. [PMID: 34715553 PMCID: PMC8642313 DOI: 10.1016/j.compbiomed.2021.104966] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2021] [Revised: 10/18/2021] [Accepted: 10/19/2021] [Indexed: 12/14/2022]
Abstract
Deep learning is a powerful tool that became practical in 2008, harnessing the power of Graphic Processing Unites, and has developed rapidly in image, video, and natural language processing. There are ongoing developments in the application of deep learning to medical data for a variety of tasks across multiple imaging modalities. The reliability and repeatability of deep learning techniques are of utmost importance if deep learning can be considered a tool for assisting experts, including physicians, radiologists, and sonographers. Owing to the high costs of labeling data, deep learning models are often evaluated against one expert, and it is unknown if any errors fall within a clinically acceptable range. Ultrasound is a commonly used imaging modality for breast cancer screening processes and for visually estimating risk using the Breast Imaging Reporting and Data System score. This process is highly dependent on the skills and experience of the sonographers and radiologists, thereby leading to interobserver variability and interpretation. For these reasons, we propose an interobserver reliability study comparing the performance of a current top-performing deep learning segmentation model against three experts who manually segmented suspicious breast lesions in clinical ultrasound (US) images. We pretrained the model using a US thyroid segmentation dataset with 455 patients and 50,993 images, and trained the model using a US breast segmentation dataset with 733 patients and 29,884 images. We found a mean Fleiss kappa value of 0.78 for the performance of three experts in breast mass segmentation compared to a mean Fleiss kappa value of 0.79 for the performance of experts and the optimized deep learning model.
Collapse
Affiliation(s)
- Jeremy M Webb
- Department of Radiology, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Shaheeda A Adusei
- Department of Physiology and Biomedical Engineering, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Yinong Wang
- Department of Radiology, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Naziya Samreen
- Department of Radiology, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Kalie Adler
- Department of Radiology, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Duane D Meixner
- Department of Radiology, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Robert T Fazzio
- Department of Radiology, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Mostafa Fatemi
- Department of Physiology and Biomedical Engineering, Mayo Clinic College of Medicine and Science, Rochester, MN
| | - Azra Alizad
- Department of Radiology, Mayo Clinic College of Medicine and Science, Rochester, MN,Department of Physiology and Biomedical Engineering, Mayo Clinic College of Medicine and Science, Rochester, MN,Corresponding author: Azra Alizad, 200 1 St. SW, Rochester, MN 55 902,
| |
Collapse
|
39
|
SSIT: a sample selection-based incremental model training method for image recognition. Neural Comput Appl 2021. [DOI: 10.1007/s00521-021-06515-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
40
|
Tajbakhsh N, Roth H, Terzopoulos D, Liang J. Guest Editorial Annotation-Efficient Deep Learning: The Holy Grail of Medical Imaging. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:2526-2533. [PMID: 34795461 PMCID: PMC8594751 DOI: 10.1109/tmi.2021.3089292] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
| | | | - Demetri Terzopoulos
- University of California, Los Angeles, and VoxelCloud, Inc., Los Angeles, CA, USA
| | | |
Collapse
|
41
|
Nath V, Yang D, Landman BA, Xu D, Roth HR. Diminishing Uncertainty Within the Training Pool: Active Learning for Medical Image Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:2534-2547. [PMID: 33373298 DOI: 10.1109/tmi.2020.3048055] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
Active learning is a unique abstraction of machine learning techniques where the model/algorithm could guide users for annotation of a set of data points that would be beneficial to the model, unlike passive machine learning. The primary advantage being that active learning frameworks select data points that can accelerate the learning process of a model and can reduce the amount of data needed to achieve full accuracy as compared to a model trained on a randomly acquired data set. Multiple frameworks for active learning combined with deep learning have been proposed, and the majority of them are dedicated to classification tasks. Herein, we explore active learning for the task of segmentation of medical imaging data sets. We investigate our proposed framework using two datasets: 1.) MRI scans of the hippocampus, 2.) CT scans of pancreas and tumors. This work presents a query-by-committee approach for active learning where a joint optimizer is used for the committee. At the same time, we propose three new strategies for active learning: 1.) increasing frequency of uncertain data to bias the training data set; 2.) Using mutual information among the input images as a regularizer for acquisition to ensure diversity in the training dataset; 3.) adaptation of Dice log-likelihood for Stein variational gradient descent (SVGD). The results indicate an improvement in terms of data reduction by achieving full accuracy while only using 22.69% and 48.85% of the available data for each dataset, respectively.
Collapse
|
42
|
Haghighi F, Taher MRH, Zhou Z, Gotway MB, Liang J. Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-Supervised Learning. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:2857-2868. [PMID: 33617450 PMCID: PMC8516596 DOI: 10.1109/tmi.2021.3060634] [Citation(s) in RCA: 26] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
This paper introduces a new concept called "transferable visual words" (TransVW), aiming to achieve annotation efficiency for deep learning in medical image analysis. Medical imaging-focusing on particular parts of the body for defined clinical purposes-generates images of great similarity in anatomy across patients and yields sophisticated anatomical patterns across images, which are associated with rich semantics about human anatomy and which are natural visual words. We show that these visual words can be automatically harvested according to anatomical consistency via self-discovery, and that the self-discovered visual words can serve as strong yet free supervision signals for deep models to learn semantics-enriched generic image representation via self-supervision (self-classification and self-restoration). Our extensive experiments demonstrate the annotation efficiency of TransVW by offering higher performance and faster convergence with reduced annotation cost in several applications. Our TransVW has several important advantages, including (1) TransVW is a fully autodidactic scheme, which exploits the semantics of visual words for self-supervised learning, requiring no expert annotation; (2) visual word learning is an add-on strategy, which complements existing self-supervised methods, boosting their performance; and (3) the learned image representation is semantics-enriched models, which have proven to be more robust and generalizable, saving annotation efforts for a variety of applications through transfer learning. Our code, pre-trained models, and curated visual words are available at https://github.com/JLiangLab/TransVW.
Collapse
|
43
|
Feng R, Zheng X, Gao T, Chen J, Wang W, Chen DZ, Wu J. Interactive Few-Shot Learning: Limited Supervision, Better Medical Image Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:2575-2588. [PMID: 33606628 DOI: 10.1109/tmi.2021.3060551] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Many known supervised deep learning methods for medical image segmentation suffer an expensive burden of data annotation for model training. Recently, few-shot segmentation methods were proposed to alleviate this burden, but such methods often showed poor adaptability to the target tasks. By prudently introducing interactive learning into the few-shot learning strategy, we develop a novel few-shot segmentation approach called Interactive Few-shot Learning (IFSL), which not only addresses the annotation burden of medical image segmentation models but also tackles the common issues of the known few-shot segmentation methods. First, we design a new few-shot segmentation structure, called Medical Prior-based Few-shot Learning Network (MPrNet), which uses only a few annotated samples (e.g., 10 samples) as support images to guide the segmentation of query images without any pre-training. Then, we propose an Interactive Learning-based Test Time Optimization Algorithm (IL-TTOA) to strengthen our MPrNet on the fly for the target task in an interactive fashion. To our best knowledge, our IFSL approach is the first to allow few-shot segmentation models to be optimized and strengthened on the target tasks in an interactive and controllable manner. Experiments on four few-shot segmentation tasks show that our IFSL approach outperforms the state-of-the-art methods by more than 20% in the DSC metric. Specifically, the interactive optimization algorithm (IL-TTOA) further contributes ~10% DSC improvement for the few-shot segmentation models.
Collapse
|
44
|
Islam NU, Gehlot S, Zhou Z, Gotway MB, Liang J. Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection. MACHINE LEARNING IN MEDICAL IMAGING. MLMI (WORKSHOP) 2021; 12966:692-702. [PMID: 35695860 PMCID: PMC9184235 DOI: 10.1007/978-3-030-87589-3_71] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Pulmonary embolism (PE) represents a thrombus ("blood clot"), usually originating from a lower extremity vein, that travels to the blood vessels in the lung, causing vascular obstruction and in some patients, death. This disorder is commonly diagnosed using CT pulmonary angiography (CTPA). Deep learning holds great promise for the computer-aided CTPA diagnosis (CAD) of PE. However, numerous competing methods for a given task in the deep learning literature exist, causing great confusion regarding the development of a CAD PE system. To address this confusion, we present a comprehensive analysis of competing deep learning methods applicable to PE diagnosis using CTPA at the both image and exam levels. At the image level, we compare convolutional neural networks (CNNs) with vision transformers, and contrast self-supervised learning (SSL) with supervised learning, followed by an evaluation of transfer learning compared with training from scratch. At the exam level, we focus on comparing conventional classification (CC) with multiple instance learning (MIL). Our extensive experiments consistently show: (1) transfer learning consistently boosts performance despite differences between natural images and CT scans, (2) transfer learning with SSL surpasses its supervised counterparts; (3) CNNs outperform vision transformers, which otherwise show satisfactory performance; and (4) CC is, surprisingly, superior to MIL. Compared with the state of the art, our optimal approach provides an AUC gain of 0.2% and 1.05% for image-level and exam-level, respectively.
Collapse
Affiliation(s)
| | - Shiv Gehlot
- Arizona State University, Tempe, AZ 85281, USA
| | | | | | | |
Collapse
|
45
|
Sun J, Liu C, Li C, Lu Z, He M, Gao L, Lin T, Sui J, Xie K, Ni X. CrossModalNet: exploiting quality preoperative images for multimodal image registration. Phys Med Biol 2021; 66. [PMID: 34330122 DOI: 10.1088/1361-6560/ac195e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2021] [Accepted: 07/30/2021] [Indexed: 11/11/2022]
Abstract
A long-standing problem in image-guided radiotherapy is that inferior intraoperative images present a difficult problem for automatic registration algorithms. Particularly for digital radiography (DR) and digitally reconstructed radiograph (DRR), the blurred, low-contrast, and noisy DR makes the multimodal registration of DR-DRR challenging. Therefore, we propose a novel CNN-based method called CrossModalNet to exploit the quality preoperative modality (DRR) for handling the limitations of intraoperative images (DR), thereby improving the registration accuracy. The method consists of two parts: DR-DRR contour predictions and contour-based rigid registration. We have designed the CrossModal Attention Module and CrossModal Refine Module to fully exploit the multiscale crossmodal features and implement the crossmodal interactions during the feature encoding and decoding stages. Then, the predicted anatomical contours of DR-DRR are registered by the classic mutual information method. We collected 2486 patient scans to train CrossModalNet and 170 scans to test its performance. The results show that it outperforms the classic and state-of-the-art methods with 95th percentile Hausdorff distance of 5.82 pixels and registration accuracy of 81.2%. The code is available at https://github.com/lc82111/crossModalNet.
Collapse
Affiliation(s)
- Jiawei Sun
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Cong Liu
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China.,Faculty of Business Information, Shanghai Business School, Shanghai 200235, People's Republic of China
| | - Chunying Li
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Zhengda Lu
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Mu He
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Liugang Gao
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Tao Lin
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Jianfeng Sui
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Kai Xie
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| | - Xinye Ni
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, People's Republic of China.,Center of Medical Physics, Nanjing Medical University, Changzhou 213003, People's Republic of China
| |
Collapse
|
46
|
Zhang Z, Li J, Tian C, Zhong Z, Jiao Z, Gao X. Quality-driven deep active learning method for 3D brain MRI segmentation. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.03.050] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
|
47
|
Ng WWY, Zhang Y, Zhang J, Wang DD, Wang FL. Stochastic Sensitivity Tree Boosting for Imbalanced Prediction Problems of Protein-Ligand Interaction Sites. IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE 2021. [DOI: 10.1109/tetci.2019.2922340] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
48
|
Hao R, Namdar K, Liu L, Khalvati F. A Transfer Learning-Based Active Learning Framework for Brain Tumor Classification. Front Artif Intell 2021; 4:635766. [PMID: 34079932 PMCID: PMC8165261 DOI: 10.3389/frai.2021.635766] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2020] [Accepted: 04/26/2021] [Indexed: 11/25/2022] Open
Abstract
Brain tumor is one of the leading causes of cancer-related death globally among children and adults. Precise classification of brain tumor grade (low-grade and high-grade glioma) at an early stage plays a key role in successful prognosis and treatment planning. With recent advances in deep learning, artificial intelligence-enabled brain tumor grading systems can assist radiologists in the interpretation of medical images within seconds. The performance of deep learning techniques is, however, highly depended on the size of the annotated dataset. It is extremely challenging to label a large quantity of medical images, given the complexity and volume of medical data. In this work, we propose a novel transfer learning-based active learning framework to reduce the annotation cost while maintaining stability and robustness of the model performance for brain tumor classification. In this retrospective research, we employed a 2D slice-based approach to train and fine-tune our model on the magnetic resonance imaging (MRI) training dataset of 203 patients and a validation dataset of 66 patients which was used as the baseline. With our proposed method, the model achieved area under receiver operating characteristic (ROC) curve (AUC) of 82.89% on a separate test dataset of 66 patients, which was 2.92% higher than the baseline AUC while saving at least 40% of labeling cost. In order to further examine the robustness of our method, we created a balanced dataset, which underwent the same procedure. The model achieved AUC of 82% compared with AUC of 78.48% for the baseline, which reassures the robustness and stability of our proposed transfer learning augmented with active learning framework while significantly reducing the size of training data.
Collapse
Affiliation(s)
- Ruqian Hao
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
- Institute of Medical Science, University of Toronto, Toronto, ON, Canada
- Department of Diagnostic Imaging, and Neurosciences and Mental Health, The Hospital for Sick Children (SickKids), Toronto, ON, Canada
| | - Khashayar Namdar
- Department of Diagnostic Imaging, and Neurosciences and Mental Health, The Hospital for Sick Children (SickKids), Toronto, ON, Canada
- Department of Medical Imaging, University of Toronto, Toronto, ON, Canada
| | - Lin Liu
- School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Farzad Khalvati
- Institute of Medical Science, University of Toronto, Toronto, ON, Canada
- Department of Diagnostic Imaging, and Neurosciences and Mental Health, The Hospital for Sick Children (SickKids), Toronto, ON, Canada
- Department of Medical Imaging, University of Toronto, Toronto, ON, Canada
- Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
49
|
Ren H, Guo W, Jiang P, Wan X. An integrated approach of Active Incremental fine-tuning, SegNet, and CRF for cutting tool wearing areas segmentation with small samples. Knowl Based Syst 2021. [DOI: 10.1016/j.knosys.2021.106838] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
50
|
StyHighNet: Semi-Supervised Learning Height Estimation from a Single Aerial Image via Unified Style Transferring. SENSORS 2021; 21:s21072272. [PMID: 33804973 PMCID: PMC8037440 DOI: 10.3390/s21072272] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/15/2021] [Revised: 03/13/2021] [Accepted: 03/21/2021] [Indexed: 11/22/2022]
Abstract
Recovering height information from a single aerial image is a key problem in the fields of computer vision and remote sensing. At present, supervised learning methods have achieved impressive results, but, due to domain bias, the trained model cannot be directly applied to a new scene. In this paper, we propose a novel semi-supervised framework, StyHighNet, for accurately estimating the height of a single aerial image in a new city that requires only a small number of labeled data. The core is to transfer multi-source images to a unified style, making the unlabeled data provide the appearance distribution as additional supervision signals. The framework mainly contains three sub-networks: (1) the style transferring sub-network maps multi-source images into unified style distribution maps (USDMs); (2) the height regression sub-network, with the function of predicting the height maps from USDMs; and (3) the style discrimination sub-network, used to distinguish the sources of USDMs. Among them, the style transferring sub-network shoulders dual responsibilities: On the one hand, it needs to compute USDMs with obvious characteristics, so that the height regression sub-network can accurately estimate the height maps. On the other hand, it is necessary that the USDMs have consistent distribution to confuse the style discrimination sub-network, so as to achieve the goal of domain adaptation. Unlike previous methods, our style distribution function is learned unsupervised, thus it is of greater flexibility and better accuracy. Furthermore, when the style discrimination sub-network is shielded, this framework can also be used for supervised learning. We performed qualitatively and quantitative evaluations on two sets of public data, Vaihingen and Potsdam. Experiments show that the framework achieved superior performance in both supervised and semi-supervised learning modes.
Collapse
|