1
|
Aghapanah H, Rasti R, Kermani S, Tabesh F, Banaem HY, Aliakbar HP, Sanei H, Segars WP. CardSegNet: An adaptive hybrid CNN-vision transformer model for heart region segmentation in cardiac MRI. Comput Med Imaging Graph 2024; 115:102382. [PMID: 38640619 DOI: 10.1016/j.compmedimag.2024.102382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2023] [Revised: 03/08/2024] [Accepted: 04/10/2024] [Indexed: 04/21/2024]
Abstract
Cardiovascular MRI (CMRI) is a non-invasive imaging technique adopted for assessing the blood circulatory system's structure and function. Precise image segmentation is required to measure cardiac parameters and diagnose abnormalities through CMRI data. Because of anatomical heterogeneity and image variations, cardiac image segmentation is a challenging task. Quantification of cardiac parameters requires high-performance segmentation of the left ventricle (LV), right ventricle (RV), and left ventricle myocardium from the background. The first proposed solution here is to manually segment the regions, which is a time-consuming and error-prone procedure. In this context, many semi- or fully automatic solutions have been proposed recently, among which deep learning-based methods have revealed high performance in segmenting regions in CMRI data. In this study, a self-adaptive multi attention (SMA) module is introduced to adaptively leverage multiple attention mechanisms for better segmentation. The convolutional-based position and channel attention mechanisms with a patch tokenization-based vision transformer (ViT)-based attention mechanism in a hybrid and end-to-end manner are integrated into the SMA. The CNN- and ViT-based attentions mine the short- and long-range dependencies for more precise segmentation. The SMA module is applied in an encoder-decoder structure with a ResNet50 backbone named CardSegNet. Furthermore, a deep supervision method with multi-loss functions is introduced to the CardSegNet optimizer to reduce overfitting and enhance the model's performance. The proposed model is validated on the ACDC2017 (n=100), M&Ms (n=321), and a local dataset (n=22) using the 10-fold cross-validation method with promising segmentation results, demonstrating its outperformance versus its counterparts.
Collapse
Affiliation(s)
- Hamed Aghapanah
- School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan, Iran
| | - Reza Rasti
- Department of Biomedical Engineering, Faculty of Engineering, University of Isfahan, Isfahan, Iran; Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA.
| | - Saeed Kermani
- School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan, Iran.
| | - Faezeh Tabesh
- Cardiovascular Research Institute, Isfahan University of Medical Sciences, Isfahan, Iran
| | - Hossein Yousefi Banaem
- Skull Base Research Center, Loghman Hakim Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Hamidreza Pour Aliakbar
- Rajaie Cardiovascular Medical and Research Center, Iran University of Medical Sciences, Tehran, Iran
| | - Hamid Sanei
- Cardiovascular Research Institute, Isfahan University of Medical Sciences, Isfahan, Iran
| | - William Paul Segars
- Department of Biomedical Engineering, Faculty of Engineering, University of Isfahan, Isfahan, Iran
| |
Collapse
|
2
|
Wang B, Ganjee R, Khandaker I, Flohr K, He Y, Li G, Wesalo J, Sahel JA, da Silva S, Pi S. Deep learning based characterization of human organoids using optical coherence tomography. BIOMEDICAL OPTICS EXPRESS 2024; 15:3112-3127. [PMID: 38855657 PMCID: PMC11161340 DOI: 10.1364/boe.515781] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/12/2023] [Revised: 03/11/2024] [Accepted: 04/08/2024] [Indexed: 06/11/2024]
Abstract
Organoids, derived from human induced pluripotent stem cells (hiPSCs), are intricate three-dimensional in vitro structures that mimic many key aspects of the complex morphology and functions of in vivo organs such as the retina and heart. Traditional histological methods, while crucial, often fall short in analyzing these dynamic structures due to their inherently static and destructive nature. In this study, we leveraged the capabilities of optical coherence tomography (OCT) for rapid, non-invasive imaging of both retinal, cerebral, and cardiac organoids. Complementing this, we developed a sophisticated deep learning approach to automatically segment the organoid tissues and their internal structures, such as hollows and chambers. Utilizing this advanced imaging and analysis platform, we quantitatively assessed critical parameters, including size, area, volume, and cardiac beating, offering a comprehensive live characterization and classification of the organoids. These findings provide profound insights into the differentiation and developmental processes of organoids, positioning quantitative OCT imaging as a potentially transformative tool for future organoid research.
Collapse
Affiliation(s)
- Bingjie Wang
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Razieh Ganjee
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Irona Khandaker
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Keevon Flohr
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Yuanhang He
- Department of Cell Biology, University of Pittsburgh, Pittsburgh, PA 15213, USA
- School of Medicine, Tsinghua University, Beijing 100084, China
| | - Guang Li
- Department of Cell Biology, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Joshua Wesalo
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - José-Alain Sahel
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Susana da Silva
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Shaohua Pi
- Department of Ophthalmology, University of Pittsburgh, Pittsburgh, PA 15213, USA
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA 15213, USA
| |
Collapse
|
3
|
Zemborain ZZ, Soifer M, Azar NS, Murillo S, Mousa HM, Perez VL, Farsiu S. Open-Source Automated Segmentation of Neuronal Structures in Corneal Confocal Microscopy Images of the Subbasal Nerve Plexus With Accuracy on Par With Human Segmentation. Cornea 2023; 42:1309-1319. [PMID: 37669422 PMCID: PMC10635613 DOI: 10.1097/ico.0000000000003319] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2022] [Accepted: 04/24/2023] [Indexed: 09/07/2023]
Abstract
PURPOSE The aim of this study was to perform automated segmentation of corneal nerves and other structures in corneal confocal microscopy (CCM) images of the subbasal nerve plexus (SNP) in eyes with ocular surface diseases (OSDs). METHODS A deep learning-based 2-stage algorithm was designed to perform segmentation of SNP features. In the first stage, to address applanation artifacts, a generative adversarial network-enabled deep network was constructed to identify 3 neighboring corneal layers on each CCM image: epithelium, SNP, and stroma. This network was trained/validated on 470 images of each layer from 73 individuals. The segmented SNP regions were further classified in the second stage by another deep network as follows: background, nerve, neuroma, and immune cells. Twenty-one-fold cross-validation was used to assess the performance of the overall algorithm on a separate data set of 207 manually segmented SNP images from 43 patients with OSD. RESULTS For the background, nerve, neuroma, and immune cell classes, the Dice similarity coefficients of the proposed automatic method were 0.992, 0.814, 0.748, and 0.736, respectively. The performance metrics for automatic segmentations were statistically better or equal as compared to human segmentation. In addition, the resulting clinical metrics had good to excellent intraclass correlation coefficients between automatic and human segmentations. CONCLUSIONS The proposed automatic method can reliably segment potential CCM biomarkers of OSD onset and progression with accuracy on par with human gradings in real clinical data, which frequently exhibited image acquisition artifacts. To facilitate future studies on OSD, we made our data set and algorithms freely available online as an open-source software package.
Collapse
Affiliation(s)
| | - Matias Soifer
- Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
- Foster Center for Ocular Immunology, Duke Eye Institute, Durham, NC, USA
| | - Nadim S. Azar
- Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
- Foster Center for Ocular Immunology, Duke Eye Institute, Durham, NC, USA
| | - Sofia Murillo
- Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
- Foster Center for Ocular Immunology, Duke Eye Institute, Durham, NC, USA
| | - Hazem M. Mousa
- Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
- Foster Center for Ocular Immunology, Duke Eye Institute, Durham, NC, USA
| | - Victor L. Perez
- Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
- Foster Center for Ocular Immunology, Duke Eye Institute, Durham, NC, USA
| | - Sina Farsiu
- Department of Biomedical Engineering, Duke University, Durham, NC, USA
- Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
| |
Collapse
|
4
|
Yang Z, Farsiu S. Directional Connectivity-based Segmentation of Medical Images. PROCEEDINGS. IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION 2023; 2023:11525-11535. [PMID: 37790907 PMCID: PMC10543919 DOI: 10.1109/cvpr52729.2023.01109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/05/2023]
Abstract
Anatomical consistency in biomarker segmentation is crucial for many medical image analysis tasks. A promising paradigm for achieving anatomically consistent segmentation via deep networks is incorporating pixel connectivity, a basic concept in digital topology, to model inter-pixel relationships. However, previous works on connectivity modeling have ignored the rich channel-wise directional information in the latent space. In this work, we demonstrate that effective disentanglement of directional sub-space from the shared latent space can significantly enhance the feature representation in the connectivity-based network. To this end, we propose a directional connectivity modeling scheme for segmentation that decouples, tracks, and utilizes the directional information across the network. Experiments on various public medical image segmentation benchmarks show the effectiveness of our model as compared to the state-of-the-art methods. Code is available at https://github.com/Zyun-Y/DconnNet.
Collapse
Affiliation(s)
- Ziyun Yang
- Duke University, Durham, NC, United States
| | | |
Collapse
|
5
|
Rasti R, Biglari A, Rezapourian M, Yang Z, Farsiu S. RetiFluidNet: A Self-Adaptive and Multi-Attention Deep Convolutional Network for Retinal OCT Fluid Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:1413-1423. [PMID: 37015695 DOI: 10.1109/tmi.2022.3228285] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
Optical coherence tomography (OCT) helps ophthalmologists assess macular edema, accumulation of fluids, and lesions at microscopic resolution. Quantification of retinal fluids is necessary for OCT-guided treatment management, which relies on a precise image segmentation step. As manual analysis of retinal fluids is a time-consuming, subjective, and error-prone task, there is increasing demand for fast and robust automatic solutions. In this study, a new convolutional neural architecture named RetiFluidNet is proposed for multi-class retinal fluid segmentation. The model benefits from hierarchical representation learning of textural, contextual, and edge features using a new self-adaptive dual-attention (SDA) module, multiple self-adaptive attention-based skip connections (SASC), and a novel multi-scale deep self-supervision learning (DSL) scheme. The attention mechanism in the proposed SDA module enables the model to automatically extract deformation-aware representations at different levels, and the introduced SASC paths further consider spatial-channel interdependencies for concatenation of counterpart encoder and decoder units, which improve representational capability. RetiFluidNet is also optimized using a joint loss function comprising a weighted version of dice overlap and edge-preserved connectivity-based losses, where several hierarchical stages of multi-scale local losses are integrated into the optimization process. The model is validated based on three publicly available datasets: RETOUCH, OPTIMA, and DUKE, with comparisons against several baselines. Experimental results on the datasets prove the effectiveness of the proposed model in retinal OCT fluid segmentation and reveal that the suggested method is more effective than existing state-of-the-art fluid segmentation algorithms in adapting to retinal OCT scans recorded by various image scanning instruments.
Collapse
|
6
|
Soltanian-Zadeh S, Liu Z, Liu Y, Lassoued A, Cukras CA, Miller DT, Hammer DX, Farsiu S. Deep learning-enabled volumetric cone photoreceptor segmentation in adaptive optics optical coherence tomography images of normal and diseased eyes. BIOMEDICAL OPTICS EXPRESS 2023; 14:815-833. [PMID: 36874491 PMCID: PMC9979662 DOI: 10.1364/boe.478693] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 01/11/2023] [Accepted: 01/12/2023] [Indexed: 06/11/2023]
Abstract
Objective quantification of photoreceptor cell morphology, such as cell diameter and outer segment length, is crucial for early, accurate, and sensitive diagnosis and prognosis of retinal neurodegenerative diseases. Adaptive optics optical coherence tomography (AO-OCT) provides three-dimensional (3-D) visualization of photoreceptor cells in the living human eye. The current gold standard for extracting cell morphology from AO-OCT images involves the tedious process of 2-D manual marking. To automate this process and extend to 3-D analysis of the volumetric data, we propose a comprehensive deep learning framework to segment individual cone cells in AO-OCT scans. Our automated method achieved human-level performance in assessing cone photoreceptors of healthy and diseased participants captured with three different AO-OCT systems representing two different types of point scanning OCT: spectral domain and swept source.
Collapse
Affiliation(s)
| | - Zhuolin Liu
- Center for Devices and Radiological Health (CDRH), U.S. Food and Drug Administration, Silver Spring, MD 20993, USA
| | - Yan Liu
- School of Optometry, Indiana University, Bloomington, IN 47405, USA
| | - Ayoub Lassoued
- School of Optometry, Indiana University, Bloomington, IN 47405, USA
| | - Catherine A. Cukras
- National Eye Institute, National Institutes of Health, Bethesda, MD 20892, USA
| | - Donald T. Miller
- School of Optometry, Indiana University, Bloomington, IN 47405, USA
| | - Daniel X. Hammer
- Center for Devices and Radiological Health (CDRH), U.S. Food and Drug Administration, Silver Spring, MD 20993, USA
| | - Sina Farsiu
- Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA
- Department of Ophthalmology, Duke University Medical Center, Durham, NC 27710, USA
| |
Collapse
|
7
|
Fan DP, Zhang J, Xu G, Cheng MM, Shao L. Salient Objects in Clutter. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:2344-2366. [PMID: 35404809 DOI: 10.1109/tpami.2022.3166451] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
In this paper, we identify and address a serious design bias of existing salient object detection (SOD) datasets, which unrealistically assume that each image should contain at least one clear and uncluttered salient object. This design bias has led to a saturation in performance for state-of-the-art SOD models when evaluated on existing datasets. However, these models are still far from satisfactory when applied to real-world scenes. Based on our analyses, we propose a new high-quality dataset and update the previous saliency benchmark. Specifically, our dataset, called Salient Objects in Clutter (SOC), includes images with both salient and non-salient objects from several common object categories. In addition to object category annotations, each salient image is accompanied by attributes that reflect common challenges in common scenes, which can help provide deeper insight into the SOD problem. Further, with a given saliency encoder, e.g., the backbone network, existing saliency models are designed to achieve mapping from the training image set to the training ground-truth set. We therefore argue that improving the dataset can yield higher performance gains than focusing only on the decoder design. With this in mind, we investigate several dataset-enhancement strategies, including label smoothing to implicitly emphasize salient boundaries, random image augmentation to adapt saliency models to various scenarios, and self-supervised learning as a regularization strategy to learn from small datasets. Our extensive results demonstrate the effectiveness of these tricks. We also provide a comprehensive benchmark for SOD, which can be found in our repository: https://github.com/DengPingFan/SODBenchmark.
Collapse
|
8
|
Wang C, Gan M. Wavelet attention network for the segmentation of layer structures on OCT images. BIOMEDICAL OPTICS EXPRESS 2022; 13:6167-6181. [PMID: 36589584 PMCID: PMC9774872 DOI: 10.1364/boe.475272] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/08/2022] [Revised: 10/21/2022] [Accepted: 10/24/2022] [Indexed: 06/17/2023]
Abstract
Automatic segmentation of layered tissue is critical for optical coherence tomography (OCT) image analysis. The development of deep learning techniques provides various solutions to this problem, while most existing methods suffer from topological errors such as outlier prediction and label disconnection. The channel attention mechanism is a powerful technique to address these problems due to its simplicity and robustness. However, it relies on global average pooling (GAP), which only calculates the lowest frequency component and leaves other potentially useful information unexplored. In this study, we use the discrete wavelet transform (DWT) to extract multi-spectral information and propose the wavelet attention network (WATNet) for tissue layer segmentation. The DWT-based attention mechanism enables multi-spectral analysis with no complex frequency-selection process and can be easily embedded to existing frameworks. Furthermore, the various wavelet bases make the WATNet adaptable to different tasks. Experiments on a self-collected esophageal dataset and two public retinal OCT dataset demonstrated that the WATNet achieved better performance compared to several widely used deep networks, confirming the advantages of the proposed method.
Collapse
Affiliation(s)
- Cong Wang
- Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
- Jinan Guoke Medical Technology Development Co., Ltd, Jinan 250102, China
| | - Meng Gan
- Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
- Jinan Guoke Medical Technology Development Co., Ltd, Jinan 250102, China
| |
Collapse
|
9
|
Dai C, Pan C, He W, Sun H. Multiple refinement and integration network for Salient Object Detection. AI COMMUN 2022. [DOI: 10.3233/aic-210273] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
The purpose of the salient object detection (SOD) task is to suppress the background noise and segment the salient foreground regions. Some previous methods considered the strategies of background suppression and multi-level feature fusion. Other methods encountered the problem that single-scale convolution features are difficult to capture the correct object size. This paper reconsiders the above problems and proposes a comprehensive solution to achieve SOD for improving the detection performance and ensuring relatively fewer parameters. First, it is difficult to achieve a better refinement effect through only one refinement operation. To this end, a multi-scale denoising module (MSDM) and multi-pooling refinement module (MPRM) are proposed to jointly complete the refinement task of multi-level features. Besides, it is difficult to fully integrate complementary features through only one feature integration operation. Mutual learning module (MLM) is proposed to preliminarily integrate multi-level features. To reduce information redundancy, multi-attention (MA) mechanism is used to assist further integration. The proposed algorithm is called multiple refinement and integration network (MRINet). Experimental results on five benchmark datasets show that MRINet outperforms state-of-the-art methods on multiple evaluation metrics. Moreover, our ResNet-based algorithm only contains 25.202 million parameters, which is less than other ResNet-based algorithms and can run at over 37 fps on a single GPU. The code will be available at https://github.com/dc3234/MRINet.
Collapse
Affiliation(s)
- Chao Dai
- Department of College of Information Engineering, China Jiliang University, China
| | - Chen Pan
- Department of College of Information Engineering, China Jiliang University, China
| | - Wei He
- Department of College of Information Engineering, China Jiliang University, China
| | - Hanqi Sun
- Department of College of Information Engineering, China Jiliang University, China
| |
Collapse
|
10
|
Yang Z, Soltanian-Zadeh S, Chu KK, Zhang H, Moussa L, Watts AE, Shaheen NJ, Wax A, Farsiu S. Connectivity-based deep learning approach for segmentation of the epithelium in in vivo human esophageal OCT images. BIOMEDICAL OPTICS EXPRESS 2021; 12:6326-6340. [PMID: 34745740 PMCID: PMC8547995 DOI: 10.1364/boe.434775] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/22/2021] [Revised: 09/01/2021] [Accepted: 09/02/2021] [Indexed: 05/03/2023]
Abstract
Optical coherence tomography (OCT) is used for diagnosis of esophageal diseases such as Barrett's esophagus. Given the large volume of OCT data acquired, automated analysis is needed. Here we propose a bilateral connectivity-based neural network for in vivo human esophageal OCT layer segmentation. Our method, connectivity-based CE-Net (Bicon-CE), defines layer segmentation as a combination of pixel connectivity modeling and pixel-wise tissue classification. Bicon-CE outperformed other widely used neural networks and reduced common topological prediction issues in tissues from healthy patients and from patients with Barrett's esophagus. This is the first end-to-end learning method developed for automatic segmentation of the epithelium in in vivo human esophageal OCT images.
Collapse
Affiliation(s)
- Ziyun Yang
- Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA
| | | | - Kengyeh K. Chu
- Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA
| | - Haoran Zhang
- Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA
| | - Lama Moussa
- Center for Esophageal Diseases and Swallowing, University of North Carolina, Chapel Hill, NC 27599, USA
| | - Ariel E. Watts
- Center for Esophageal Diseases and Swallowing, University of North Carolina, Chapel Hill, NC 27599, USA
| | - Nicholas J. Shaheen
- Center for Esophageal Diseases and Swallowing, University of North Carolina, Chapel Hill, NC 27599, USA
| | - Adam Wax
- Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA
| | - Sina Farsiu
- Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA
- Department of Ophthalmology, Duke University School of Medicine, Durham, NC 27710, USA
| |
Collapse
|