UNesT: Local spatial representation learning with hierarchical transformer for efficient medical segmentation

Leveraging segmentation-guided spatial feature embedding for overall survival prediction in glioblastoma with multimodal magnetic resonance imaging

BACKGROUND AND OBJECTIVE

Patients with glioblastoma have a five-year relative survival rate of less than 5 %. Thus, accurately predicting the overall survival (OS) of patients with glioblastoma is crucial for effective treatment planning.

METHODS

To fully leverage the imaging characteristics of glioblastomas, we propose a segmentation-guided regression method for predicting OS of patients with brain tumors using multimodal magnetic resonance imaging. Specifically, a brain tumor segmentation network was first pre-trained without leveraging survival information. Subsequently, the survival regression network was jointly trained with the guidance of brain tumor segmentation, focusing on tumor voxels and suppressing irrelevant backgrounds.

RESULTS

Our proposed framework, based on the well-known backbone of UNETR++, achieved a Dice score of 0.7910, Spearman correlation of 0.4112, and Harrell's concordance index of 0.6488. The model consistently showed promising results compared with baseline methods on two different datasets (BraTS and UCSF-PDGM). Furthermore, ablation studies on our training configurations demonstrated that both the pre-training segmentation network and contrastive loss significantly improved all metrics for OS prediction.

CONCLUSIONS

In this study, we propose a joint learning framework based on a pre-trained segmentation backbone for OS prediction by leveraging a brain tumor segmentation map. By utilizing a spatial feature map, our model can operate using a sliding-window approach, which can be adopted by varying the matrix sizes and resolutions of the input images.

Segmenting medical images with limited data

While computer vision has proven valuable for medical image segmentation, its application faces challenges such as limited dataset sizes and the complexity of effectively leveraging unlabeled images. To address these challenges, we present a novel semi-supervised, consistency-based approach termed the data-efficient medical segmenter (DEMS). The DEMS features an encoder-decoder architecture and incorporates the developed online automatic augmenter (OAA) and residual robustness enhancement (RRE) blocks. The OAA augments input data with various image transformations, thereby diversifying the dataset to improve the generalization ability. The RRE enriches feature diversity and introduces perturbations to create varied inputs for different decoders, thereby providing enhanced variability. Moreover, we introduce a sensitive loss to further enhance consistency across different decoders and stabilize the training process. Extensive experimental results on both our own and three public datasets affirm the effectiveness of DEMS. Under extreme data shortage scenarios, our DEMS achieves 16.85% and 10.37% improvement in dice score compared with the U-Net and top-performed state-of-the-art method, respectively. Given its superior data efficiency, DEMS could present significant advancements in medical segmentation under small data regimes. The project homepage can be accessed at https://github.com/NUS-Tim/DEMS.

Towards accurate abdominal tumor segmentation: A 2D model with Position-Aware and Key Slice Feature Sharing

Abdominal tumor segmentation is a crucial yet challenging step during the screening and diagnosis of tumors. While 3D segmentation models provide powerful performance, they demand substantial computational resources. Additionally, in 3D data, tumors often represent a small portion, leading to imbalanced data and potentially overlooking crucial information. Conversely, 2D segmentation models have a lightweight structure, but disregard the inter-slice correlation, risking the loss of tumor in edge slices. To address these challenges, this paper proposes a novel Position-Aware and Key Slice Feature Sharing 2D tumor segmentation model (PAKS-Net). Leveraging the Swin-Transformer, we effectively model the global features within each slice, facilitating essential information extraction. Furthermore, we introduce a Position-Aware module to capture the spatial relationship between tumors and their corresponding organs, mitigating noise and interference from surrounding organ tissues. To enhance the edge slice segmentation accuracy, we employ key slices to assist in the segmentation of other slices to prioritize tumor regions. Through extensive experiments on three abdominal tumor segmentation CT datasets and a lung tumor segmentation CT dataset, PAKS-Net demonstrates superior performance, reaching 0.893, 0.769, 0.598 and 0.738 tumor DSC on the KiTS19, LiTS17, pancreas and LOTUS datasets, surpassing 3D segmentation models, while remaining computationally efficient with fewer parameters.

A mixed Mamba U-net for prostate segmentation in MR images

The diagnosis of early prostate cancer depends on the accurate segmentation of prostate regions in magnetic resonance imaging (MRI). However, this segmentation task is challenging due to the particularities of prostate MR images themselves and the limitations of existing methods. To address these issues, we propose a U-shaped encoder-decoder network MM-UNet based on Mamba and CNN for prostate segmentation in MR images. Specifically, we first proposed an adaptive feature fusion module based on channel attention guidance to achieve effective fusion between adjacent hierarchical features and suppress the interference of background noise. Secondly, we propose a global context-aware module based on Mamba, which has strong long-range modeling capabilities and linear complexity, to capture global context information in images. Finally, we propose a multi-scale anisotropic convolution module based on the principle of parallel multi-scale anisotropic convolution blocks and 3D convolution decomposition. Experimental results on two public prostate MR image segmentation datasets demonstrate that the proposed method outperforms competing models in terms of prostate segmentation performance and achieves state-of-the-art performance. In future work, we intend to enhance the model's robustness and extend its applicability to additional medical image segmentation tasks.

Associations between the choroid plexus and tau in Alzheimer's disease using an active learning segmentation pipeline

BACKGROUND

The cerebrospinal fluid (CSF), primarily generated by the choroid plexus (ChP), is the major carrier of the glymphatic system. The alternations of CSF production and the ChP can be associated with the Alzheimer's disease (AD). The present work investigated the roles of the ChP in the AD based on a proposed ChP image segmentation pipeline.

METHODS

A human-in-the-loop ChP image segmentation pipeline was implemented with intermediate and active learning datasets. The performance of the proposed pipeline was evaluated on manual contours by five radiologists, compared to the FreeSurfer and FastSurfer toolboxes. The ChP volume and blood flow were investigated among AD groups. The correlations between the ChP volume and AD CSF biomarkers including phosphorylated tau (p-tau), total tau (t-tau), amyloid-β42 (Aβ42), and amyloid-β40 (Aβ40) was investigated using three models (univariate, multiple variables, and stepwise regression) on two datasets with 806 and 320 subjects.

RESULTS

The proposed ChP segmentation pipeline achieved superior performance with a Dice coefficient of 0.620 on the test dataset, compared to the FreeSurfer (0.342) and FastSurfer (0.371). Significantly larger volumes (p < 0.001) and higher perfusion (p = 0.032) at the ChP were found in AD compared to CN groups. Significant correlations were found between the tau and the relative ChP volume (the ChP volume and ChP/parenchyma ratio) in each patient groups and in the univariate regression analysis (p < 0.001), the multiple regression model (p < 0.05 except for the t-tau in the LMCI), and in the step-wise regression model (p < 0.021). In addition, the correlation coefficients changed from - 0.32 to - 0.21 along with the AD progression in the multiple regression model. In contrast, the Aβ42 and Aβ40 shows consistent and significant associations with the lateral ventricle related measures in the step-wise regression model (p < 0.027).

CONCLUSIONS

The proposed pipeline provided accurate ChP segmentation which revealed the associations between the ChP and tau level in the AD. The proposed pipeline is available on GitHub ( https://github.com/princeleeee/ChP-Seg ).

A novel LVPA-UNet network for target volume automatic delineation: An MRI case study of nasopharyngeal carcinoma

Accurate delineation of Gross Tumor Volume (GTV) is crucial for radiotherapy. Deep learning-driven GTV segmentation technologies excel in rapidly and accurately delineating GTV, providing a basis for radiologists in formulating radiation plans. The existing 2D and 3D segmentation models of GTV based on deep learning are limited by the loss of spatial features and anisotropy respectively, and are both affected by the variability of tumor characteristics, blurred boundaries, and background interference. All these factors seriously affect the segmentation performance. To address the above issues, a Layer-Volume Parallel Attention (LVPA)-UNet model based on 2D-3D architecture has been proposed in this study, in which three strategies are introduced. Firstly, 2D and 3D workflows are introduced in the LVPA-UNet. They work in parallel and can guide each other. Both the fine features of each slice of 2D MRI and the 3D anatomical structure and spatial features of the tumor can be extracted by them. Secondly, parallel multi-branch depth-wise strip convolutions adapt the model to tumors of varying shapes and sizes within slices and volumetric spaces, and achieve refined processing of blurred boundaries. Lastly, a Layer-Channel Attention mechanism is proposed to adaptively adjust the weights of slices and channels according to their different tumor information, and then to highlight slices and channels with tumor. The experiments by LVPA-UNet on 1010 nasopharyngeal carcinoma (NPC) MRI datasets from three centers show a DSC of 0.7907, precision of 0.7929, recall of 0.8025, and HD95 of 1.8702 mm, outperforming eight typical models. Compared to the baseline model, it improves DSC by 2.14 %, precision by 2.96 %, and recall by 1.01 %, while reducing HD95 by 0.5434 mm. Consequently, while ensuring the efficiency of segmentation through deep learning, LVPA-UNet is able to provide superior GTV delineation results for radiotherapy and offer technical support for precision medicine.

Glioma segmentation based on dense contrastive learning and multimodal features recalibration

Accurate segmentation of different regions of gliomas from multimodal magnetic resonance (MR) images is crucial for glioma grading and precise diagnosis, but many existing segmentation methods are difficult to effectively utilize multimodal MR image information to recognize accurately the lesion regions with small size, low contrast and irregular shape. To address this issue, this work proposes a novel 3D glioma segmentation model DCL-MANet. DCL-MANet has an architecture of multiple encoders and one single decoder. Each encoder is used to extract MR image features of a given modality. To overcome the entangle problems of multimodal semantic features, a dense contrastive learning (DCL) strategy is presented to extract the modality-specific and common features. Following that, feature recalibration block (RFB) based on modality-wise attention is used to recalibrate the semantic features of each modality, enabling the model to focus on the features that are beneficial for glioma segmentation. These recalibrated features are input into the decoder to obtain the segmentation results. To verify the superiority of the proposed method, we compare it with several state-of-the-art (SOTA) methods in terms of Dice, average symmetric surface distance (ASSD), HD95 and volumetric similarity (Vs). The comparison results show that the average Dice, ASSD, HD95 and Vs of DCL-MANet on all tumor regions are improved at least by 0.66%, 3.47%, 8.94% and 1.07% respectively. For small enhance tumor (ET) region, the corresponding improvement can be up to 0.37%, 7.83%, 11.32%, and 1.35%, respectively. In addition, the ablation results demonstrate the effectiveness of the proposed DCL and RFB, and combining them can significantly increase Dice (1.59%) and Vs (1.54%) while decreasing ASSD (40.51%) and HD95 (45.16%) on ET region. The proposed DCL-MANet could disentangle multimodal features and enhance the semantics of modality-dependent features, providing a potential means to accurately segment small lesion regions in gliomas.

WET-UNet: Wavelet integrated efficient transformer networks for nasopharyngeal carcinoma tumor segmentation

Nasopharyngeal carcinoma is a malignant tumor that occurs in the epithelium and mucosal glands of the nasopharynx, and its pathological type is mostly poorly differentiated squamous cell carcinoma. Since the nasopharynx is located deep in the head and neck, early diagnosis and timely treatment are critical to patient survival. However, nasopharyngeal carcinoma tumors are small in size and vary widely in shape, and it is also a challenge for experienced doctors to delineate tumor contours. In addition, due to the special location of nasopharyngeal carcinoma, complex treatments such as radiotherapy or surgical resection are often required, so accurate pathological diagnosis is also very important for the selection of treatment options. However, the current deep learning segmentation model faces the problems of inaccurate segmentation and unstable segmentation process, which are mainly limited by the accuracy of data sets, fuzzy boundaries, and complex lines. In order to solve these two challenges, this article proposes a hybrid model WET-UNet based on the UNet network as a powerful alternative for nasopharyngeal cancer image segmentation. On the one hand, wavelet transform is integrated into UNet to enhance the lesion boundary information by using low-frequency components to adjust the encoder at low frequencies and optimize the subsequent computational process of the Transformer to improve the accuracy and robustness of image segmentation. On the other hand, the attention mechanism retains the most valuable pixels in the image for us, captures the remote dependencies, and enables the network to learn more representative features to improve the recognition ability of the model. Comparative experiments show that our network structure outperforms other models for nasopharyngeal cancer image segmentation, and we demonstrate the effectiveness of adding two modules to help tumor segmentation. The total data set of this article is 5000, and the ratio of training and verification is 8:2. In the experiment, accuracy = 85.2% and precision = 84.9% can show that our proposed model has good performance in nasopharyngeal cancer image segmentation.

Application of visual transformer in renal image analysis

Deep Self-Attention Network (Transformer) is an encoder-decoder architectural model that excels in establishing long-distance dependencies and is first applied in natural language processing. Due to its complementary nature with the inductive bias of convolutional neural network (CNN), Transformer has been gradually applied to medical image processing, including kidney image processing. It has become a hot research topic in recent years. To further explore new ideas and directions in the field of renal image processing, this paper outlines the characteristics of the Transformer network model and summarizes the application of the Transformer-based model in renal image segmentation, classification, detection, electronic medical records, and decision-making systems, and compared with CNN-based renal image processing algorithm, analyzing the advantages and disadvantages of this technique in renal image processing. In addition, this paper gives an outlook on the development trend of Transformer in renal image processing, which provides a valuable reference for a lot of renal image analysis.

Deep learning segmentation of organs-at-risk with integration into clinical workflow for pediatric brain radiotherapy

PURPOSE

Radiation therapy (RT) of pediatric brain cancer is known to be associated with long-term neurocognitive deficits. Although target and organs-at-risk (OARs) are contoured as part of treatment planning, other structures linked to cognitive functions are often not included. This paper introduces a novel automatic segmentation tool specifically designed for the unique challenges posed by pediatric patients undergoing brain RT, as well as its seamless integration into the existing clinical workflow.

METHODS AND MATERIALS

Images of 47 pediatric brain cancer patients aged 1 to 20 years old and 33 two-year-old healthy infants were used to train a vision transformer, UNesT, for the segmentation of five brain OARs. The trained model was then incorporated to clinical workflow via DICOM connections between a treatment planning system (TPS) and a server hosting the trained model such that scans are sent from TPS to the server, automatically segmented, and sent back to TPS for treatment planning.

RESULTS

The proposed automatic segmentation framework achieved a median dice similarity coefficient of 0.928 (frontal white matter), 0.908 (corpus callosum), 0.933 (hippocampi), 0.819 (temporal lobes), and 0.960 (brainstem) with a mean ± SD run time of 1.8 ± 0.67 s over 20 test cases.

CONCLUSIONS

The pediatric brain segmentation tool showed promising performance on five OARs linked to neurocognitive functions and can easily be extended for additional structures. The proposed integration to the clinic enables easy access to the tool from clinical platforms and minimizes disruption to existing workflow while maximizing its benefits.

Enhancing Hierarchical Transformers for Whole Brain Segmentation with Intracranial Measurements Integration

Whole brain segmentation with magnetic resonance imaging (MRI) enables the non-invasive measurement of brain regions, including total intracranial volume (TICV) and posterior fossa volume (PFV). Enhancing the existing whole brain segmentation methodology to incorporate intracranial measurements offers a heightened level of comprehensiveness in the analysis of brain structures. Despite its potential, the task of generalizing deep learning techniques for intracranial measurements faces data availability constraints due to limited manually annotated atlases encompassing whole brain and TICV/PFV labels. In this paper, we enhancing the hierarchical transformer UNesT for whole brain segmentation to achieve segmenting whole brain with 133 classes and TICV/PFV simultaneously. To address the problem of data scarcity, the model is first pretrained on 4859 T1-weighted (T1w) 3D volumes sourced from 8 different sites. These volumes are processed through a multi-atlas segmentation pipeline for label generation, while TICV/PFV labels are unavailable. Subsequently, the model is finetuned with 45 T1w 3D volumes from Open Access Series Imaging Studies (OASIS) where both 133 whole brain classes and TICV/PFV labels are available. We evaluate our method with Dice similarity coefficients(DSC). We show that our model is able to conduct precise TICV/PFV estimation while maintaining the 132 brain regions performance at a comparable level. Code and trained model are available at: https://github.com/MASILab/UNesT/wholebrainSeg.

Development of the next-generation functional neuro-cognitive imaging protocol - Part 1: A 3D sliding-window convolutional neural net for automated brain parcellation

Functional MRI has emerged as a powerful tool to assess the severity of Post-concussion syndrome (PCS) and to provide guidance for neuro-cognitive therapists during treatment. The next-generation functional neuro-cognitive imaging protocol (fNCI2) has been developed to provide this assessment. This paper covers the first step in the analysis process, the development of a rapidly re-trainable, machine-learning, brain parcellation tool. The use of a sufficiently deep U-Net architecture encompassing a small (39 × 39 × 39 voxel input, 27 × 27 × 27 voxel output) sliding window to sample the entirety of the 3D image allows for the prediction of the entire image using only a single trained network. A large number of training, validating, and testing windows are thus generated from the 101 manually-labeled Mindboggle images, and full-image prediction is provided via a voxel-vote method using overlapping windows. Our method produces parcellated images that are highly consistent with standard atlas-based methods in under 3 min on a modern GPU, and the single network architecture allows for rapid retraining (<36 hr) as needed.