Loss odyssey in medical image segmentation

View adaptive unified self-supervised technique for abdominal organ segmentation

Automatic abdominal organ segmentation is an essential prerequisite for accurate volumetric analysis, disease diagnosis, and tracking by medical practitioners. However, the deformable shapes, variable locations, overlapping with nearby organs, and similar contrast make the segmentation challenging. Moreover, the requirement of a large manually labeled dataset makes it harder. Hence, a semi-supervised contrastive learning approach is utilized to perform the automatic abdominal organ segmentation. Existing 3D deep learning models based on contrastive learning are not able to capture the 3D context of medical volumetric data along three planes/views: axial, sagittal, and coronal views. In this work, a semi-supervised view-adaptive unified model (VAU-model) is proposed to make the 3D deep learning model as view-adaptive to learn 3D context along each view in a unified manner. This method utilizes the novel optimization function that assists the 3D model to learn the 3D context of volumetric medical data along each view in a single model. The effectiveness of the proposed approach is validated on the three types of datasets: BTCV, NIH, and MSD quantitatively and qualitatively. The results demonstrate that the VAU model achieves an average Dice score of 81.61% which is a 3.89% improvement compared to the previous best results for pancreas segmentation in multi-organ dataset BTCV. It also achieves an average Dice score of 77.76% and 76.76% for the pancreas under the single organ non-pathological NIH dataset, and pathological MSD dataset.

Towards more precise automatic analysis: a systematic review of deep learning-based multi-organ segmentation

Accurate segmentation of multiple organs in the head, neck, chest, and abdomen from medical images is an essential step in computer-aided diagnosis, surgical navigation, and radiation therapy. In the past few years, with a data-driven feature extraction approach and end-to-end training, automatic deep learning-based multi-organ segmentation methods have far outperformed traditional methods and become a new research topic. This review systematically summarizes the latest research in this field. We searched Google Scholar for papers published from January 1, 2016 to December 31, 2023, using keywords "multi-organ segmentation" and "deep learning", resulting in 327 papers. We followed the PRISMA guidelines for paper selection, and 195 studies were deemed to be within the scope of this review. We summarized the two main aspects involved in multi-organ segmentation: datasets and methods. Regarding datasets, we provided an overview of existing public datasets and conducted an in-depth analysis. Concerning methods, we categorized existing approaches into three major classes: fully supervised, weakly supervised and semi-supervised, based on whether they require complete label information. We summarized the achievements of these methods in terms of segmentation accuracy. In the discussion and conclusion section, we outlined and summarized the current trends in multi-organ segmentation.

Positional contrastive learning for improved thigh muscle segmentation in MR images

The accurate segmentation of individual muscles is essential for quantitative MRI analysis of thigh images. Deep learning methods have achieved state-of-the-art results in segmentation, but they require large numbers of labeled data to perform well. However, labeling individual thigh muscles slice by slice for numerous volumes is a laborious and time-consuming task, which limits the availability of annotated datasets. To address this challenge, self-supervised learning (SSL) emerges as a promising technique to enhance model performance by pretraining the model on unlabeled data. A recent approach, called positional contrastive learning, exploits the information given by the axial position of the slices to learn features transferable on the segmentation task. The aim of this work was to propose positional contrastive SSL for the segmentation of individual thigh muscles from MRI acquisitions in a population of elderly healthy subjects and to evaluate it on different levels of limited annotated data. An unlabeled dataset of 72 T1w MRI thigh acquisitions was available for SSL pretraining, while a labeled dataset of 52 volumes was employed for the final segmentation task, split into training and test sets. The effectiveness of SSL pretraining to fine-tune a U-Net architecture for thigh muscle segmentation was compared with that of a randomly initialized model (RND), considering an increasing number of annotated volumes (S = 1, 2, 5, 10, 20, 30, 40). Our results demonstrated that SSL yields substantial improvements in Dice similarity coefficient (DSC) when using a very limited number of labeled volumes (e.g., forS $$ S $$  = 1, DSC 0.631 versus 0.530 for SSL and RND, respectively). Moreover, enhancements are achievable even when utilizing the full number of labeled subjects, with DSC = 0.927 for SSL and 0.924 for RND. In conclusion, positional contrastive SSL was effective in obtaining more accurate thigh muscle segmentation, even with a very low number of labeled data, with a potential impact of speeding up the annotation process in clinics.

Uncertainty estimation- and attention-based semi-supervised models for automatically delineate clinical target volume in CBCT images of breast cancer

OBJECTIVES

Accurate segmentation of the clinical target volume (CTV) of CBCT images can observe the changes of CTV during patients' radiotherapy, and lay a foundation for the subsequent implementation of adaptive radiotherapy (ART). However, segmentation is challenging due to the poor quality of CBCT images and difficulty in obtaining target volumes. An uncertainty estimation- and attention-based semi-supervised model called residual convolutional block attention-uncertainty aware mean teacher (RCBA-UAMT) was proposed to delineate the CTV in cone-beam computed tomography (CBCT) images of breast cancer automatically.

METHODS

A total of 60 patients who undergone radiotherapy after breast-conserving surgery were enrolled in this study, which involved 60 planning CTs and 380 CBCTs. RCBA-UAMT was proposed by integrating residual and attention modules in the backbone network 3D UNet. The attention module can adjust channel and spatial weights of the extracted image features. The proposed design can train the model and segment CBCT images with a small amount of labeled data (5%, 10%, and 20%) and a large amount of unlabeled data. Four types of evaluation metrics, namely, dice similarity coefficient (DSC), Jaccard, average surface distance (ASD), and 95% Hausdorff distance (95HD), are used to assess the model segmentation performance quantitatively.

RESULTS

The proposed method achieved average DSC, Jaccard, 95HD, and ASD of 82%, 70%, 8.93, and 1.49 mm for CTV delineation on CBCT images of breast cancer, respectively. Compared with the three classical methods of mean teacher, uncertainty-aware mean-teacher and uncertainty rectified pyramid consistency, DSC and Jaccard increased by 7.89-9.33% and 14.75-16.67%, respectively, while 95HD and ASD decreased by 33.16-67.81% and 36.05-75.57%, respectively. The comparative experiment results of the labeled data with different proportions (5%, 10% and 20%) showed significant differences in the DSC, Jaccard, and 95HD evaluation indexes in the labeled data with 5% versus 10% and 5% versus 20%. Moreover, no significant differences were observed in the labeled data with 10% versus 20% among all evaluation indexes. Therefore, we can use only 10% labeled data to achieve the experimental objective.

CONCLUSIONS

Using the proposed RCBA-UAMT, the CTV of breast cancer CBCT images can be delineated reliably with a small amount of labeled data. These delineated images can be used to observe the changes in CTV and lay the foundation for the follow-up implementation of ART.

Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation

Precise delineation of multiple organs or abnormal regions in the human body from medical images plays an essential role in computer-aided diagnosis, surgical simulation, image-guided interventions, and especially in radiotherapy treatment planning. Thus, it is of great significance to explore automatic segmentation approaches, among which deep learning-based approaches have evolved rapidly and witnessed remarkable progress in multi-organ segmentation. However, obtaining an appropriately sized and fine-grained annotated dataset of multiple organs is extremely hard and expensive. Such scarce annotation limits the development of high-performance multi-organ segmentation models but promotes many annotation-efficient learning paradigms. Among these, studies on transfer learning leveraging external datasets, semi-supervised learning including unannotated datasets and partially-supervised learning integrating partially-labeled datasets have led the dominant way to break such dilemmas in multi-organ segmentation. We first review the fully supervised method, then present a comprehensive and systematic elaboration of the 3 abovementioned learning paradigms in the context of multi-organ segmentation from both technical and methodological perspectives, and finally summarize their challenges and future trends.

A novel approach to the detection of facial wrinkles: Database, detection algorithm, and evaluation metrics

Skin wrinkles result from intrinsic aging processes and extrinsic influences, including prolonged exposure to ultraviolet radiation and tobacco smoking. Hence, the identification of wrinkles holds significant importance in skin aging and medical aesthetic investigation. Nevertheless, current methods lack the comprehensiveness to identify facial wrinkles, particularly those that may appear insignificant. Furthermore, the current assessment techniques neglect to consider the blurred boundary of wrinkles and cannot differentiate images with varying resolutions. This research introduces a novel wrinkle detection algorithm and a distance-based loss function to identify full-face wrinkles. Furthermore, we develop a wrinkle detection evaluation metric that assesses outcomes based on curve, location, and gradient similarity. We collected and annotated a dataset for wrinkle detection consisting of 1021 images of Chinese faces. The dataset will be made publicly available to further promote wrinkle detection research. The research demonstrates a substantial enhancement in detecting subtle wrinkles through implementing the proposed method. Furthermore, the suggested evaluation procedure effectively considers the indistinct boundaries of wrinkles and is applicable to images with various resolutions.

Linear semantic transformation for semi-supervised medical image segmentation

Medical image segmentation is a focus research and foundation in developing intelligent medical systems. Recently, deep learning for medical image segmentation has become a standard process and succeeded significantly, promoting the development of reconstruction, and surgical planning of disease diagnosis. However, semantic learning is often inefficient owing to the lack of supervision of feature maps, resulting in that high-quality segmentation models always rely on numerous and accurate data annotations. Learning robust semantic representation in latent spaces remains a challenge. In this paper, we propose a novel semi-supervised learning framework to learn vital attributes in medical images, which constructs generalized representation from diverse semantics to realize medical image segmentation. We first build a self-supervised learning part that achieves context recovery by reconstructing space and intensity of medical images, which conduct semantic representation for feature maps. Subsequently, we combine semantic-rich feature maps and utilize simple linear semantic transformation to convert them into image segmentation. The proposed framework was tested using five medical segmentation datasets. Quantitative assessments indicate the highest scores of our method on IXI (73.78%), ScaF (47.50%), COVID-19-Seg (50.72%), PC-Seg (65.06%), and Brain-MR (72.63%) datasets. Finally, we compared our method with the latest semi-supervised learning methods and obtained 77.15% and 75.22% DSC values, respectively, ranking first on two representative datasets. The experimental results not only proved that the proposed linear semantic transformation was effectively applied to medical image segmentation, but also presented its simplicity and ease-of-use to pursue robust segmentation in semi-supervised learning. Our code is now open at: https://github.com/QingYunA/Linear-Semantic-Transformation-for-Semi-Supervised-Medical-Image-Segmentation.

A new family of instance-level loss functions for improving instance-level segmentation and detection of white matter hyperintensities in routine clinical brain MRI

In this study, we introduce "instance loss functions", a new family of loss functions designed to enhance the training of neural networks in the instance-level segmentation and detection of objects in biomedical image data, particularly those of varied numbers and sizes. Intended to be utilized conjointly with traditional loss functions, these proposed functions, prioritize object instances over pixel-by-pixel comparisons. The specific functions, the instance segmentation loss (Linstance), the instance center loss (Lcenter), the false instance rate loss (Lfalse), and the instance proximity loss (Lproximity), serve distinct purposes. Specifically, Linstance improves instance-wise segmentation quality, Lcenter enhances segmentation quality of small instances, Lfalse minimizes the rate of false and missed detections across varied instance sizes, and Lproximity improves detection quality by pulling predicted instances towards the ground truth instances. Through the task of segmenting white matter hyperintensities (WMH) in brain MRI, we benchmarked our proposed instance loss functions, both individually and in combination via an ensemble inference models approach, against traditional pixel-level loss functions. Data were sourced from the Alzheimer's Disease Neuroimaging Initiative (ADNI) and the WMH Segmentation Challenge datasets, which exhibit significant variation in WMH instance sizes. Empirical evaluations demonstrate that combining two instance-level loss functions through ensemble inference models outperforms models using other loss function on both the ADNI and WMH Segmentation Challenge datasets for the segmentation and detection of WMH instances. Further, applying these functions to the segmentation of nuclei in histopathology images demonstrated their effectiveness and generalizability beyond WMH, improving performance even in contexts with less severe instance imbalance.

A Novel Mis-Seg-Focus Loss Function Based on a Two-Stage nnU-Net Framework for Accurate Brain Tissue Segmentation

Brain tissue segmentation plays a critical role in the diagnosis, treatment, and study of brain diseases. Accurately identifying these boundaries is essential for improving segmentation accuracy. However, distinguishing boundaries between different brain tissues can be challenging, as they often overlap. Existing deep learning methods primarily calculate the overall segmentation results without adequately addressing local regions, leading to error propagation and mis-segmentation along boundaries. In this study, we propose a novel mis-segmentation-focused loss function based on a two-stage nnU-Net framework. Our approach aims to enhance the model's ability to handle ambiguous boundaries and overlapping anatomical structures, thereby achieving more accurate brain tissue segmentation results. Specifically, the first stage targets the identification of mis-segmentation regions using a global loss function, while the second stage involves defining a mis-segmentation loss function to adaptively adjust the model, thus improving its capability to handle ambiguous boundaries and overlapping anatomical structures. Experimental evaluations on two datasets demonstrate that our proposed method outperforms existing approaches both quantitatively and qualitatively.

SALW-Net: a lightweight convolutional neural network based on self-adjusting loss function for spine MR image segmentation

Segmentation of intervertebral discs and vertebrae from spine magnetic resonance (MR) images is essential to aid diagnosis algorithms for lumbar disc herniation. Convolutional neural networks (CNN) are effective methods, but often require high computational costs. Designing a lightweight CNN is more suitable for medical sites lacking high-computing power devices, yet due to the unbalanced pixel distribution in spine MR images, the segmentation is often sub-optimal. To address this issue, a lightweight spine segmentation CNN based on a self-adjusting loss function, which is named SALW-Net, is proposed in this study. For SALW-Net, the self-adjusting loss function could dynamically adjust the loss weights of the two branches according to the differences in segmentation results and labels during the training; thus, the ability for learning unbalanced pixels is enhanced. Two separate datasets are used to evaluate the proposed SALW-Net. Specifically, the proposed SALW-Net has fewer parameter numbers than U-net (only 2%) but achieves higher evaluation scores than that of U-net (the average DSC score of SALW-Net is 0.8781, and that of U-net is 0.8482). In addition, the practicality validation for SALW-Net is also proceeding, including deploying the model on a lightweight device and producing an aid diagnosis algorithm based on segmentation results. This means our SALW-Net has clinical application potential for assisted diagnosis in low computational power scenarios.

Automated quantification of vacuole fusion and lipophagy in Saccharomyces cerevisiae from fluorescence and cryo-soft X-ray microscopy data using deep learning

During starvation in the yeast Saccharomyces cerevisiae vacuolar vesicles fuse and lipid droplets (LDs) can become internalized into the vacuole in an autophagic process named lipophagy. There is a lack of tools to quantitatively assess starvation-induced vacuole fusion and lipophagy in intact cells with high resolution and throughput. Here, we combine soft X-ray tomography (SXT) with fluorescence microscopy and use a deep-learning computational approach to visualize and quantify these processes in yeast. We focus on yeast homologs of mammalian NPC1 (NPC intracellular cholesterol transporter 1; Ncr1 in yeast) and NPC2 proteins, whose dysfunction leads to Niemann Pick type C (NPC) disease in humans. We developed a convolutional neural network (CNN) model which classifies fully fused versus partially fused vacuoles based on fluorescence images of stained cells. This CNN, named Deep Yeast Fusion Network (DYFNet), revealed that cells lacking Ncr1 (ncr1∆ cells) or Npc2 (npc2∆ cells) have a reduced capacity for vacuole fusion. Using a second CNN model, we implemented a pipeline named LipoSeg to perform automated instance segmentation of LDs and vacuoles from high-resolution reconstructions of X-ray tomograms. From that, we obtained 3D renderings of LDs inside and outside of the vacuole in a fully automated manner and additionally measured droplet volume, number, and distribution. We find that ncr1∆ and npc2∆ cells could ingest LDs into vacuoles normally but showed compromised degradation of LDs and accumulation of lipid vesicles inside vacuoles. Our new method is versatile and allows for analysis of vacuole fusion, droplet size and lipophagy in intact cells.Abbreviations: BODIPY493/503: 4,4-difluoro-1,3,5,7,8-pentamethyl-4-bora-3a,4a-diaza-s-Indacene; BPS: bathophenanthrolinedisulfonic acid disodium salt hydrate; CNN: convolutional neural network; DHE; dehydroergosterol; npc2∆, yeast deficient in Npc2; DSC, Dice similarity coefficient; EM, electron microscopy; EVs, extracellular vesicles; FIB-SEM, focused ion beam milling-scanning electron microscopy; FM 4-64, N-(3-triethylammoniumpropyl)-4-(6-[4-{diethylamino} phenyl] hexatrienyl)-pyridinium dibromide; LDs, lipid droplets; Ncr1, yeast homolog of human NPC1 protein; ncr1∆, yeast deficient in Ncr1; NPC, Niemann Pick type C; NPC2, Niemann Pick type C homolog; OD600, optical density at 600 nm; ReLU, rectifier linear unit; PPV, positive predictive value; NPV, negative predictive value; MCC, Matthews correlation coefficient; SXT, soft X-ray tomography; UV, ultraviolet; YPD, yeast extract peptone dextrose.

A Lightweight Convolutional Neural Network Based on Dynamic Level-Set Loss Function for Spine MR Image Segmentation

BACKGROUND

Spine MR image segmentation is important foundation for computer-aided diagnostic (CAD) algorithms of spine disorders. Convolutional neural networks segment effectively, but require high computational costs.

PURPOSE

To design a lightweight model based on dynamic level-set loss function for high segmentation performance.

STUDY TYPE

Retrospective.

POPULATION

Four hundred forty-eight subjects (3163 images) from two separate datasets. Dataset-1: 276 subjects/994 images (53.26% female, mean age 49.02 ± 14.09), all for disc degeneration screening, 188 had disc degeneration, 67 had herniated disc. Dataset-2: public dataset with 172 subjects/2169 images, 142 patients with vertebral degeneration, 163 patients with disc degeneration.

FIELD STRENGTH/SEQUENCE

T2 weighted turbo spin echo sequences at 3T.

ASSESSMENT

Dynamic Level-set Net (DLS-Net) was compared with four mainstream (including U-net++) and four lightweight models, and manual label made by five radiologists (vertebrae, discs, spinal fluid) used as segmentation evaluation standard. Five-fold cross-validation are used for all experiments. Based on segmentation, a CAD algorithm of lumbar disc was designed for assessing DLS-Net's practicality, and the text annotation (normal, bulging, or herniated) from medical history data were used as evaluation standard.

STATISTICAL TESTS

All segmentation models were evaluated with DSC, accuracy, precision, and AUC. The pixel numbers of segmented results were compared with manual label using paired t-tests, with P < 0.05 indicating significance. The CAD algorithm was evaluated with accuracy of lumbar disc diagnosis.

RESULTS

With only 1.48% parameters of U-net++, DLS-Net achieved similar accuracy in both datasets (Dataset-1: DSC 0.88 vs. 0.89, AUC 0.94 vs. 0.94; Dataset-2: DSC 0.86 vs. 0.86, AUC 0.93 vs. 0.93). The segmentation results of DLS-Net showed no significant differences with manual labels in pixel numbers for discs (Dataset-1: 1603.30 vs. 1588.77, P = 0.22; Dataset-2: 863.61 vs. 886.4, P = 0.14) and vertebrae (Dataset-1: 3984.28 vs. 3961.94, P = 0.38; Dataset-2: 4806.91 vs. 4732.85, P = 0.21). Based on DLS-Net's segmentation results, the CAD algorithm achieved higher accuracy than using non-cropped MR images (87.47% vs. 61.82%).

DATA CONCLUSION

The proposed DLS-Net has fewer parameters but achieves similar accuracy to U-net++, helps CAD algorithm achieve higher accuracy, which facilitates wider application.

EVIDENCE LEVEL

2 TECHNICAL EFFICACY: Stage 1.

Semi-supervised learning towards automated segmentation of PET images with limited annotations: application to lymphoma patients

Manual segmentation poses a time-consuming challenge for disease quantification, therapy evaluation, treatment planning, and outcome prediction. Convolutional neural networks (CNNs) hold promise in accurately identifying tumor locations and boundaries in PET scans. However, a major hurdle is the extensive amount of supervised and annotated data necessary for training. To overcome this limitation, this study explores semi-supervised approaches utilizing unlabeled data, specifically focusing on PET images of diffuse large B-cell lymphoma (DLBCL) and primary mediastinal large B-cell lymphoma (PMBCL) obtained from two centers. We considered 2-[18F]FDG PET images of 292 patients PMBCL (n = 104) and DLBCL (n = 188) (n = 232 for training and validation, and n = 60 for external testing). We harnessed classical wisdom embedded in traditional segmentation methods, such as the fuzzy clustering loss function (FCM), to tailor the training strategy for a 3D U-Net model, incorporating both supervised and unsupervised learning approaches. Various supervision levels were explored, including fully supervised methods with labeled FCM and unified focal/Dice loss, unsupervised methods with robust FCM (RFCM) and Mumford-Shah (MS) loss, and semi-supervised methods combining FCM with supervised Dice loss (MS + Dice) or labeled FCM (RFCM + FCM). The unified loss function yielded higher Dice scores (0.73 ± 0.11; 95% CI 0.67-0.8) than Dice loss (p value < 0.01). Among the semi-supervised approaches, RFCM + αFCM (α = 0.3) showed the best performance, with Dice score of 0.68 ± 0.10 (95% CI 0.45-0.77), outperforming MS + αDice for any supervision level (any α) (p < 0.01). Another semi-supervised approach with MS + αDice (α = 0.2) achieved Dice score of 0.59 ± 0.09 (95% CI 0.44-0.76) surpassing other supervision levels (p < 0.01). Given the time-consuming nature of manual delineations and the inconsistencies they may introduce, semi-supervised approaches hold promise for automating medical imaging segmentation workflows.

CA-UNet Segmentation Makes a Good Ischemic Stroke Risk Prediction

Stroke is still the World's second major factor of death, as well as the third major factor of death and disability. Ischemic stroke is a type of stroke, in which early detection and treatment are the keys to preventing ischemic strokes. However, due to the limitation of privacy protection and labeling difficulties, there are only a few studies on the intelligent automatic diagnosis of stroke or ischemic stroke, and the results are unsatisfactory. Therefore, we collect some data and propose a 3D carotid Computed Tomography Angiography (CTA) image segmentation model called CA-UNet for fully automated extraction of carotid arteries. We explore the number of down-sampling times applicable to carotid segmentation and design a multi-scale loss function to resolve the loss of detailed features during the process of down-sampling. Moreover, based on CA-Unet, we propose an ischemic stroke risk prediction model to predict the risk in patients using their 3D CTA images, electronic medical records, and medical history. We have validated the efficacy of our segmentation model and prediction model through comparison tests. Our method can provide reliable diagnoses and results that benefit patients and medical professionals.

Navigating the nuances: comparative analysis and hyperparameter optimisation of neural architectures on contrast-enhanced MRI for liver and liver tumour segmentation

In medical imaging, accurate segmentation is crucial to improving diagnosis, treatment, or both. However, navigating the multitude of available architectures for automatic segmentation can be overwhelming, making it challenging to determine the appropriate type of architecture and tune the most crucial parameters during dataset optimisation. To address this problem, we examined and refined seven distinct architectures for segmenting the liver, as well as liver tumours, with a restricted training collection of 60 3D contrast-enhanced magnetic resonance images (CE-MRI) from the ATLAS dataset. Included in these architectures are convolutional neural networks (CNNs), transformers, and hybrid CNN/transformer architectures. Bayesian search techniques were used for hyperparameter tuning to hasten convergence to the optimal parameter mixes while also minimising the number of trained models. It was unexpected that hybrid models, which typically exhibit superior performance on larger datasets, would exhibit comparable performance to CNNs. The optimisation of parameters contributed to better segmentations, resulting in an average increase of 1.7% and 5.0% in liver and tumour segmentation Dice coefficients, respectively. In conclusion, the findings of this study indicate that hybrid CNN/transformer architectures may serve as a practical substitute for CNNs even in small datasets. This underscores the significance of hyperparameter optimisation.

Semi-supervised image segmentation using a residual-driven mean teacher and an exponential Dice loss

Semi-supervised segmentation plays an important role in computer vision and medical image analysis and can alleviate the burden of acquiring abundant expert-annotated images. In this paper, we developed a residual-driven semi-supervised segmentation method (termed RDMT) based on the classical mean teacher (MT) framework by introducing a novel model-level residual perturbation and an exponential Dice (eDice) loss. The introduced perturbation was integrated into the exponential moving average (EMA) scheme to enhance the performance of the MT, while the eDice loss was used to improve the detection sensitivity of a given network to object boundaries. We validated the developed method by applying it to segment 3D Left Atrium (LA) and 2D optic cup (OC) from the public LASC and REFUGE datasets based on the V-Net and U-Net, respectively. Extensive experiments demonstrated that the developed method achieved the average Dice score of 0.8776 and 0.7751, when trained on 10% and 20% labeled images, respectively for the LA and OC regions depicted on the LASC and REFUGE datasets. It significantly outperformed the MT and can compete with several existing semi-supervised segmentation methods (i.e., HCMT, UAMT, DTC and SASS).

Piscis: a novel loss estimator of the F1 score enables accurate spot detection in fluorescence microscopy images via deep learning

Single-molecule RNA fluorescence in situ hybridization (RNA FISH)-based spatial transcriptomics methods have enabled the accurate quantification of gene expression at single-cell resolution by visualizing transcripts as diffraction-limited spots. While these methods generally scale to large samples, image analysis remains challenging, often requiring manual parameter tuning. We present Piscis, a fully automatic deep learning algorithm for spot detection trained using a novel loss function, the SmoothF1 loss, that approximates the F1 score to directly penalize false positives and false negatives but remains differentiable and hence usable for training by deep learning approaches. Piscis was trained and tested on a diverse dataset composed of 358 manually annotated experimental RNA FISH images representing multiple cell types and 240 additional synthetic images. Piscis outperforms other state-of-the-art spot detection methods, enabling accurate, high-throughput analysis of RNA FISH-derived imaging data without the need for manual parameter tuning.

Segment anything in medical images

Medical image segmentation is a critical component in clinical practice, facilitating accurate diagnosis, treatment planning, and disease monitoring. However, existing methods, often tailored to specific modalities or disease types, lack generalizability across the diverse spectrum of medical image segmentation tasks. Here we present MedSAM, a foundation model designed for bridging this gap by enabling universal medical image segmentation. The model is developed on a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types. We conduct a comprehensive evaluation on 86 internal validation tasks and 60 external validation tasks, demonstrating better accuracy and robustness than modality-wise specialist models. By delivering accurate and efficient segmentation across a wide spectrum of tasks, MedSAM holds significant potential to expedite the evolution of diagnostic tools and the personalization of treatment plans.

Real-time reliable semantic segmentation of thyroid nodules in ultrasound images

Objective.Low efficiency in medical image segmentation is a common issue that limits computer-aided diagnosis development. Due to the varying positions and sizes of nodules, it is not easy to accurately segment ultrasound images. This study aims to propose a segmentation model that maintains high efficiency while improving accuracy.Approach. We propose a novel layer that integrates the advantages of dense connectivity, dilated convolution, and factorized filters to maintain excellent efficiency while improving accuracy. Dense connectivity optimizes feature reuse, dilated convolution redesigns layers, and factorized convolution improves efficiency. Moreover, we propose a loss function optimization method from a pixel perspective to increase the network's accuracy further.Main results.Experiments on the Thyroid dataset show that our method achieves 81.70% intersection-over-union (IoU), 90.50% true positive rate (TPR), and 0.25% false positive rate (FPR). In terms of accuracy, our method outperforms the state-of-the-art methods, with twice faster inference and nearly 400 times fewer parameters. Meanwhile, in a test on an External Thyroid dataset, our method achieves 77.03% IoU, 82.10% TPR, and 0.16% FPR, demonstrating our proposed model's robustness.Significance.We propose a real-time semantic segmentation architecture for thyroid nodule segmentation in ultrasound images called fully convolution dense dilated network (FCDDN). Our method runs fast with a few parameters and is suitable for medical devices requiring real-time segmentation.

An interactive nuclei segmentation framework with Voronoi diagrams and weighted convex difference for cervical cancer pathology images

Objective.Nuclei segmentation is crucial for pathologists to accurately classify and grade cancer. However, this process faces significant challenges, such as the complex background structures in pathological images, the high-density distribution of nuclei, and cell adhesion.Approach.In this paper, we present an interactive nuclei segmentation framework that increases the precision of nuclei segmentation. Our framework incorporates expert monitoring to gather as much prior information as possible and accurately segment complex nucleus images through limited pathologist interaction, where only a small portion of the nucleus locations in each image are labeled. The initial contour is determined by the Voronoi diagram generated from the labeled points, which is then input into an optimized weighted convex difference model to regularize partition boundaries in an image. Specifically, we provide theoretical proof of the mathematical model, stating that the objective function monotonically decreases. Furthermore, we explore a postprocessing stage that incorporates histograms, which are simple and easy to handle and prevent arbitrariness and subjectivity in individual choices.Main results.To evaluate our approach, we conduct experiments on both a cervical cancer dataset and a nasopharyngeal cancer dataset. The experimental results demonstrate that our approach achieves competitive performance compared to other methods.Significance.The Voronoi diagram in the paper serves as prior information for the active contour, providing positional information for individual cells. Moreover, the active contour model achieves precise segmentation results while offering mathematical interpretability.

DDCNN-F: double decker convolutional neural network 'F' feature fusion as a medical image classification framework

Melanoma is a severe skin cancer that involves abnormal cell development. This study aims to provide a new feature fusion framework for melanoma classification that includes a novel 'F' Flag feature for early detection. This novel 'F' indicator efficiently distinguishes benign skin lesions from malignant ones known as melanoma. The article proposes an architecture that is built in a Double Decker Convolutional Neural Network called DDCNN future fusion. The network's deck one, known as a Convolutional Neural Network (CNN), finds difficult-to-classify hairy images using a confidence factor termed the intra-class variance score. These hirsute image samples are combined to form a Baseline Separated Channel (BSC). By eliminating hair and using data augmentation techniques, the BSC is ready for analysis. The network's second deck trains the pre-processed BSC and generates bottleneck features. The bottleneck features are merged with features generated from the ABCDE clinical bio indicators to promote classification accuracy. Different types of classifiers are fed to the resulting hybrid fused features with the novel 'F' Flag feature. The proposed system was trained using the ISIC 2019 and ISIC 2020 datasets to assess its performance. The empirical findings expose that the DDCNN feature fusion strategy for exposing malignant melanoma achieved a specificity of 98.4%, accuracy of 93.75%, precision of 98.56%, and Area Under Curve (AUC) value of 0.98. This study proposes a novel approach that can accurately identify and diagnose fatal skin cancer and outperform other state-of-the-art techniques, which is attributed to the DDCNN 'F' Feature fusion framework. Also, this research ascertained improvements in several classifiers when utilising the 'F' indicator, resulting in the highest specificity of + 7.34%.

Adaptive t-vMF dice loss: An effective expansion of dice loss for medical image segmentation

Dice loss is widely used for medical image segmentation, and many improved loss functions have been proposed. However, further Dice loss improvements are still possible. In this study, we reconsidered the use of Dice loss and discovered that Dice loss can be rewritten in the loss function using the cosine similarity through a simple equation transformation. Using this knowledge, we present a novel t-vMF Dice loss based on the t-vMF similarity instead of the cosine similarity. Based on the t-vMF similarity, our proposed Dice loss is formulated in a more compact similarity loss function than the original Dice loss. Furthermore, we present an effective algorithm that automatically determines the parameter κ for the t-vMF similarity using a validation accuracy, called Adaptive t-vMF Dice loss. Using this algorithm, it is possible to apply more compact similarities for easy classes and wider similarities for difficult classes, and we are able to achieve adaptive training based on the accuracy of each class. We evaluated binary segmentation datasets of CVC-ClinicDB and Kvasir-SEG, and multi-class segmentation datasets of Automated Cardiac Diagnosis Challenge and Synapse multi-organ segmentation. Through experiments conducted on four datasets using a five-fold cross-validation, we confirmed that the Dice score coefficient (DSC) was further improved in comparison with the original Dice loss and other loss functions.

Multi-scale context UNet-like network with redesigned skip connections for medical image segmentation

BACKGROUND AND OBJECTIVE

Medical image segmentation has garnered significant research attention in the neural network community as a fundamental requirement for developing intelligent medical assistant systems. A series of UNet-like networks with an encoder-decoder architecture have achieved remarkable success in medical image segmentation. Among these networks, UNet2+ (UNet++) and UNet3+ (UNet+++) have introduced redesigned skip connections, dense skip connections, and full-scale skip connections, respectively, surpassing the performance of the original UNet. However, UNet2+ lacks comprehensive information obtained from the entire scale, which hampers its ability to learn organ placement and boundaries. Similarly, due to the limited number of neurons in its structure, UNet3+ fails to effectively segment small objects when trained with a small number of samples.

METHOD

In this study, we propose UNet_sharp (UNet#), a novel network topology named after the "#" symbol, which combines dense skip connections and full-scale skip connections. In the decoder sub-network, UNet# can effectively integrate feature maps of different scales and capture fine-grained features and coarse-grained semantics from the entire scale. This approach enhances the understanding of organ and lesion positions and enables accurate boundary segmentation. We employ deep supervision for model pruning to accelerate testing and enable mobile device deployment. Additionally, we construct two classification-guided modules to reduce false positives and improve segmentation accuracy.

RESULTS

Compared to current UNet-like networks, our proposed method achieves the highest Intersection over Union (IoU) values ((92.67±0.96)%, (92.38±1.29)%, (95.36±1.22)%, (74.01±2.03)%) and F1 scores ((91.64±1.86)%, (95.70±2.16)%, (97.34±2.76)%, (84.77±2.65)%) on the semantic segmentation tasks of nuclei, brain tumors, liver, and lung nodules, respectively.

CONCLUSIONS

The experimental results demonstrate that the reconstructed skip connections in UNet successfully incorporate multi-scale contextual semantic information. Compared to most state-of-the-art medical image segmentation models, our proposed method more accurately locates organs and lesions and precisely segments boundaries.

Do we really need dice? The hidden region-size biases of segmentation losses

Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses. On the surface, these two categories of losses (i.e., distribution based vs. geometry based) seem unrelated, and there is no clear consensus as to which category is a better choice, with varying performances for each across different benchmarks and applications. Furthermore, it is widely argued within the medical-imaging community that Dice and CE are complementary, which has motivated the use of compound CE-Dice losses. In this work, we provide a theoretical analysis, which shows that CE and Dice share a much deeper connection than previously thought. First, we show that, from a constrained-optimization perspective, they both decompose into two components, i.e., a similar ground-truth matching term, which pushes the predicted foreground regions towards the ground-truth, and a region-size penalty term imposing different biases on the size (or proportion) of the predicted regions. Then, we provide bound relationships and an information-theoretic analysis, which uncover hidden region-size biases: Dice has an intrinsic bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. It also explains why CE dominates natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting to different region-size distributions. Based on our theoretical analysis, we propose a principled and simple solution, which enables to control explicitly the region-size bias. The proposed method integrates CE with explicit terms based on L1 or the KL divergence, which encourage segmenting region proportions to match target class proportions, thereby mitigating class imbalance but without losing generality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of explicit and simple region-size terms. The code is available at https://github.com/by-liu/SegLossBias .

Assessment of heart-substructures auto-contouring accuracy for application in heart-sparing radiotherapy for lung cancer

Objectives

We validated an auto-contouring algorithm for heart substructures in lung cancer patients, aiming to establish its accuracy and reliability for radiotherapy (RT) planning. We focus on contouring an amalgamated set of subregions in the base of the heart considered to be a new organ at risk, the cardiac avoidance area (CAA), to enable maximum dose limit implementation in lung RT planning.

Methods

The study validates a deep-learning model specifically adapted for auto-contouring the CAA (which includes the right atrium, aortic valve root, and proximal segments of the left and right coronary arteries). Geometric, dosimetric, quantitative, and qualitative validation measures are reported. Comparison with manual contours, including assessment of interobserver variability, and robustness testing over 198 cases are also conducted.

Results

Geometric validation shows that auto-contouring performance lies within the expected range of manual observer variability despite being slightly poorer than the average of manual observers (mean surface distance for CAA of 1.6 vs 1.2 mm, dice similarity coefficient of 0.86 vs 0.88). Dosimetric validation demonstrates consistency between plans optimized using auto-contours and manual contours. Robustness testing confirms acceptable contours in all cases, with 80% rated as "Good" and the remaining 20% as "Useful."

Conclusions

The auto-contouring algorithm for heart substructures in lung cancer patients demonstrates acceptable and comparable performance to human observers.

Advances in knowledge

Accurate and reliable auto-contouring results for the CAA facilitate the implementation of a maximum dose limit to this region in lung RT planning, which has now been introduced in the routine setting at our institution.

Expanded tube attention for tubular structure segmentation

PURPOSE

Semantic segmentation of tubular structures, such as blood vessels and cell membranes, is a very difficult task, and it tends to break many predicted regions in the middle. This problem is due to the fact that tubular ground truth is very thin, and the number of pixels is extremely unbalanced compared to the background.

METHODS

We present a novel training method using pseudo-labels generated by morphological transformation. Furthermore, we present an attention module using thickened pseudo-labels, called the expanded tube attention (ETA) module. By using the ETA module, the network learns thickened regions based on pseudo-labels at first and then gradually learns thinned original regions while transferring information in the thickened regions as an attention map.

RESULTS

Through experiments conducted on retina vessel image datasets using various evaluation measures, we confirmed that the proposed method using ETA modules improved the clDice metric accuracy in comparison with the conventional methods.

CONCLUSIONS

We demonstrated that the proposed novel expanded tube attention module using thickened pseudo-labels can achieve easy-to-hard learning.

Adaptive Region-Specific Loss for Improved Medical Image Segmentation

Defining the loss function is an important part of neural network design and critically determines the success of deep learning modeling. A significant shortcoming of the conventional loss functions is that they weight all regions in the input image volume equally, despite the fact that the system is known to be heterogeneous (i.e., some regions can achieve high prediction performance more easily than others). Here, we introduce a region-specific loss to lift the implicit assumption of homogeneous weighting for better learning. We divide the entire volume into multiple sub-regions, each with an individualized loss constructed for optimal local performance. Effectively, this scheme imposes higher weightings on the sub-regions that are more difficult to segment, and vice versa. Furthermore, the regional false positive and false negative errors are computed for each input image during a training step and the regional penalty is adjusted accordingly to enhance the overall accuracy of the prediction. Using different public and in-house medical image datasets, we demonstrate that the proposed regionally adaptive loss paradigm outperforms conventional methods in the multi-organ segmentations, without any modification to the neural network architecture or additional data preparation.

Ceph-Net: automatic detection of cephalometric landmarks on scanned lateral cephalograms from children and adolescents using an attention-based stacked regression network

BACKGROUND

The success of cephalometric analysis depends on the accurate detection of cephalometric landmarks on scanned lateral cephalograms. However, manual cephalometric analysis is time-consuming and can cause inter- and intra-observer variability. The purpose of this study was to automatically detect cephalometric landmarks on scanned lateral cephalograms with low contrast and resolution using an attention-based stacked regression network (Ceph-Net).

METHODS

The main body of Ceph-Net compromised stacked fully convolutional networks (FCN) which progressively refined the detection of cephalometric landmarks on each FCN. By embedding dual attention and multi-path convolution modules in Ceph-Net, the network learned local and global context and semantic relationships between cephalometric landmarks. Additionally, the intermediate deep supervision in each FCN further boosted the training stability and the detection performance of cephalometric landmarks.

RESULTS

Ceph-Net showed a superior detection performance in mean radial error and successful detection rate, including accuracy improvements in cephalometric landmark detection located in low-contrast soft tissues compared with other detection networks. Moreover, Ceph-Net presented superior detection performance on the test dataset split by age from 8 to 16 years old.

CONCLUSIONS

Ceph-Net demonstrated an automatic and superior detection of cephalometric landmarks by successfully learning local and global context and semantic relationships between cephalometric landmarks in scanned lateral cephalograms with low contrast and resolutions.

Improved UNet with Attention for Medical Image Segmentation

Medical image segmentation is crucial for medical image processing and the development of computer-aided diagnostics. In recent years, deep Convolutional Neural Networks (CNNs) have been widely adopted for medical image segmentation and have achieved significant success. UNet, which is based on CNNs, is the mainstream method used for medical image segmentation. However, its performance suffers owing to its inability to capture long-range dependencies. Transformers were initially designed for Natural Language Processing (NLP), and sequence-to-sequence applications have demonstrated the ability to capture long-range dependencies. However, their abilities to acquire local information are limited. Hybrid architectures of CNNs and Transformer, such as TransUNet, have been proposed to benefit from Transformer's long-range dependencies and CNNs' low-level details. Nevertheless, automatic medical image segmentation remains a challenging task due to factors such as blurred boundaries, the low-contrast tissue environment, and in the context of ultrasound, issues like speckle noise and attenuation. In this paper, we propose a new model that combines the strengths of both CNNs and Transformer, with network architectural improvements designed to enrich the feature representation captured by the skip connections and the decoder. To this end, we devised a new attention module called Three-Level Attention (TLA). This module is composed of an Attention Gate (AG), channel attention, and spatial normalization mechanism. The AG preserves structural information, whereas channel attention helps to model the interdependencies between channels. Spatial normalization employs the spatial coefficient of the Transformer to improve spatial attention akin to TransNorm. To further improve the skip connection and reduce the semantic gap, skip connections between the encoder and decoder were redesigned in a manner similar to that of the UNet++ dense connection. Moreover, deep supervision using a side-output channel was introduced, analogous to BASNet, which was originally used for saliency predictions. Two datasets from different modalities, a CT scan dataset and an ultrasound dataset, were used to evaluate the proposed UNet architecture. The experimental results showed that our model consistently improved the prediction performance of the UNet across different datasets.

PMJAF-Net: Pyramidal multi-scale joint attention and adaptive fusion network for explainable skin lesion segmentation

Traditional convolutional neural networks have achieved remarkable success in skin lesion segmentation. However, the successive pooling operations and convolutional spans reduce the feature resolution and hinder the dense prediction for spatial information, resulting in blurred boundaries, low accuracy and poor interpretability for irregular lesion segmentation under low contrast. To solve the above issues, a pyramidal multi-scale joint attention and adaptive fusion network for explainable (PMJAF-Net) skin lesion segmentation is proposed. Firstly, an adaptive spatial attention module is designed to establish the long-term correlation between pixels, enrich the global and local contextual information, and refine the detailed features. Subsequently, an efficient pyramidal multi-scale channel attention module is proposed to capture the multi-scale information and edge features by using the pyramidal module. Meanwhile, a channel attention module is devised to establish the long-term correlation between channels and highlight the most related feature channels to capture the multi-scale key information on each channel. Thereafter, a multi-scale adaptive fusion attention module is put forward to efficiently fuse the scale features at different decoding stages. Finally, a novel hybrid loss function based on region salient features and boundary quality is presented to guide the network to learn from map-level, patch-level and pixel-level and to accurately predict the lesion regions with clear boundaries. In addition, visualizing attention weight maps are utilized to visually enhance the interpretability of our proposed model. Comprehensive experiments are conducted on four public skin lesion datasets, and the results demonstrate that the proposed network outperforms the state-of-the-art methods, with the segmentation assessment evaluation metrics Dice, JI, and ACC improved to 92.65%, 87.86% and 96.26%, respectively.

Deep learning-assisted identification and quantification of aneurysmal subarachnoid hemorrhage in non-contrast CT scans: Development and external validation of Hybrid 2D/3D UNet

Accurate stroke assessment and consequent favorable clinical outcomes rely on the early identification and quantification of aneurysmal subarachnoid hemorrhage (aSAH) in non-contrast computed tomography (NCCT) images. However, hemorrhagic lesions can be complex and difficult to distinguish manually. To solve these problems, here we propose a novel Hybrid 2D/3D UNet deep-learning framework for automatic aSAH identification and quantification in NCCT images. We evaluated 1824 consecutive patients admitted with aSAH to four hospitals in China between June 2018 and May 2022. Accuracy and precision, Dice scores and intersection over union (IoU), and interclass correlation coefficients (ICC) were calculated to assess model performance, segmentation performance, and correlations between automatic and manual segmentation, respectively. A total of 1355 patients with aSAH were enrolled: 931, 101, 179, and 144 in four datasets, of whom 326 were scanned with Siemens, 640 with Philips, and 389 with GE Medical Systems scanners. Our proposed deep-learning method accurately identified (accuracies 0.993-0.999) and segmented (Dice scores 0.550-0.897) hemorrhage in both the internal and external datasets, even combinations of hemorrhage subtypes. We further developed a convenient AI-assisted platform based on our algorithm to assist clinical workflows, whose performance was comparable to manual measurements by experienced neurosurgeons (ICCs 0.815-0.957) but with greater efficiency and reduced cost. While this tool has not yet been prospectively tested in clinical practice, our innovative hybrid network algorithm and platform can accurately identify and quantify aSAH, paving the way for fast and cheap NCCT interpretation and a reliable AI-based approach to expedite clinical decision-making for aSAH patients.

Fibroglandular tissue segmentation in breast MRI using vision transformers: a multi-institutional evaluation

Accurate and automatic segmentation of fibroglandular tissue in breast MRI screening is essential for the quantification of breast density and background parenchymal enhancement. In this retrospective study, we developed and evaluated a transformer-based neural network for breast segmentation (TraBS) in multi-institutional MRI data, and compared its performance to the well established convolutional neural network nnUNet. TraBS and nnUNet were trained and tested on 200 internal and 40 external breast MRI examinations using manual segmentations generated by experienced human readers. Segmentation performance was assessed in terms of the Dice score and the average symmetric surface distance. The Dice score for nnUNet was lower than for TraBS on the internal testset (0.909 ± 0.069 versus 0.916 ± 0.067, P < 0.001) and on the external testset (0.824 ± 0.144 versus 0.864 ± 0.081, P = 0.004). Moreover, the average symmetric surface distance was higher (= worse) for nnUNet than for TraBS on the internal (0.657 ± 2.856 versus 0.548 ± 2.195, P = 0.001) and on the external testset (0.727 ± 0.620 versus 0.584 ± 0.413, P = 0.03). Our study demonstrates that transformer-based networks improve the quality of fibroglandular tissue segmentation in breast MRI compared to convolutional-based models like nnUNet. These findings might help to enhance the accuracy of breast density and parenchymal enhancement quantification in breast MRI screening.

Bridged-U-Net-ASPP-EVO and Deep Learning Optimization for Brain Tumor Segmentation

Brain tumor segmentation from Magnetic Resonance Images (MRI) is considered a big challenge due to the complexity of brain tumor tissues, and segmenting these tissues from the healthy tissues is an even more tedious challenge when manual segmentation is undertaken by radiologists. In this paper, we have presented an experimental approach to emphasize the impact and effectiveness of deep learning elements like optimizers and loss functions towards a deep learning optimal solution for brain tumor segmentation. We evaluated our performance results on the most popular brain tumor datasets (MICCAI BraTS 2020 and RSNA-ASNR-MICCAI BraTS 2021). Furthermore, a new Bridged U-Net-ASPP-EVO was introduced that exploits Atrous Spatial Pyramid Pooling to enhance capturing multi-scale information to help in segmenting different tumor sizes, Evolving Normalization layers, squeeze and excitation residual blocks, and the max-average pooling for down sampling. Two variants of this architecture were constructed (Bridged U-Net_ASPP_EVO v1 and Bridged U-Net_ASPP_EVO v2). The best results were achieved using these two models when compared with other state-of-the-art models; we have achieved average segmentation dice scores of 0.84, 0.85, and 0.91 from variant1, and 0.83, 0.86, and 0.92 from v2 for the Enhanced Tumor (ET), Tumor Core (TC), and Whole Tumor (WT) tumor sub-regions, respectively, in the BraTS 2021validation dataset.

Automated T1 and T2 mapping segmentation on cardiovascular magnetic resonance imaging using deep learning

Introduction

Structural and functional heart abnormalities can be examined non-invasively with cardiac magnetic resonance imaging (CMR). Thanks to the development of MR devices, diagnostic scans can capture more and more relevant information about possible heart diseases. T1 and T2 mapping are such novel technology, providing tissue specific information even without the administration of contrast material. Artificial intelligence solutions based on deep learning have demonstrated state-of-the-art results in many application areas, including medical imaging. More specifically, automated tools applied at cine sequences have revolutionized volumetric CMR reporting in the past five years. Applying deep learning models to T1 and T2 mapping images can similarly improve the efficiency of post-processing pipelines and consequently facilitate diagnostic processes.

Methods

In this paper, we introduce a deep learning model for myocardium segmentation trained on over 7,000 raw CMR images from 262 subjects of heterogeneous disease etiology. The data were labeled by three experts. As part of the evaluation, Dice score and Hausdorff distance among experts is calculated, and the expert consensus is compared with the model's predictions.

Results

Our deep learning method achieves 86% mean Dice score, while contours provided by three experts on the same data show 90% mean Dice score. The method's accuracy is consistent across epicardial and endocardial contours, and on basal, midventricular slices, with only 5% lower results on apical slices, which are often challenging even for experts.

Conclusions

We trained and evaluated a deep learning based segmentation model on 262 heterogeneous CMR cases. Applying deep neural networks to T1 and T2 mapping could similarly improve diagnostic practices. Using the fine details of T1 and T2 mapping images and high-quality labels, the objective of this research is to approach human segmentation accuracy with deep learning.

BiDFDC-Net: a dense connection network based on bi-directional feedback for skin image segmentation

Accurate segmentation of skin lesions in dermoscopic images plays an important role in improving the survival rate of patients. However, due to the blurred boundaries of pigment regions, the diversity of lesion features, and the mutations and metastases of diseased cells, the effectiveness and robustness of skin image segmentation algorithms are still a challenging subject. For this reason, we proposed a bi-directional feedback dense connection network framework (called BiDFDC-Net), which can perform skin lesions accurately. Firstly, under the framework of U-Net, we integrated the edge modules into each layer of the encoder which can solve the problem of gradient vanishing and network information loss caused by network deepening. Then, each layer of our model takes input from the previous layer and passes its feature map to the densely connected network of subsequent layers to achieve information interaction and enhance feature propagation and reuse. Finally, in the decoder stage, a two-branch module was used to feed the dense feedback branch and the ordinary feedback branch back to the same layer of coding, to realize the fusion of multi-scale features and multi-level context information. By testing on the two datasets of ISIC-2018 and PH2, the accuracy on the two datasets was given by 93.51% and 94.58%, respectively.

Predicting DWI-FLAIR mismatch on NCCT: the role of artificial intelligence in hyperacute decision making

Background

The presence of diffusion-weighted imaging (DWI) and fluid-attenuated inversion recovery (FLAIR) mismatch was used to determine eligibility for intravenous thrombolysis in clinical trials. However, due to the restricted availability of MRI and the ambiguity of image assessment, it is not widely implemented in clinical practice.

Methods

A total of 222 acute ischemic stroke patients underwent non-contrast computed tomography (NCCT), DWI, and FLAIR within 1 h of one another. Human experts manually segmented ischemic lesions on DWI and FLAIR images and independently graded the presence of DWI-FLAIR mismatch. Deep learning (DL) models based on the nnU-net architecture were developed to predict ischemic lesions visible on DWI and FLAIR images using NCCT images. Inexperienced neurologists evaluated the DWI-FLAIR mismatch on NCCT images without and with the model's results.

Results

The mean age of included subjects was 71.8 ± 12.8  years, 123 (55%) were male, and the baseline NIHSS score was a median of 11 [IQR, 6-18]. All images were taken in the following order: NCCT - DWI - FLAIR, starting after a median of 139 [81-326] min after the time of the last known well. Intravenous thrombolysis was administered in 120 patients (54%) after NCCT. The DL model's prediction on NCCT images revealed a Dice coefficient and volume correlation of 39.1% and 0.76 for DWI lesions and 18.9% and 0.61 for FLAIR lesions. In the subgroup with 15  mL or greater lesion volume, the evaluation of DWI-FLAIR mismatch from NCCT by inexperienced neurologists improved in accuracy (from 0.537 to 0.610) and AUC-ROC (from 0.493 to 0.613).

Conclusion

The DWI-FLAIR mismatch may be reckoned using NCCT images through advanced artificial intelligence techniques.

A survey on deep learning for skin lesion segmentation

Skin cancer is a major public health problem that could benefit from computer-aided diagnosis to reduce the burden of this common disease. Skin lesion segmentation from images is an important step toward achieving this goal. However, the presence of natural and artificial artifacts (e.g., hair and air bubbles), intrinsic factors (e.g., lesion shape and contrast), and variations in image acquisition conditions make skin lesion segmentation a challenging task. Recently, various researchers have explored the applicability of deep learning models to skin lesion segmentation. In this survey, we cross-examine 177 research papers that deal with deep learning-based segmentation of skin lesions. We analyze these works along several dimensions, including input data (datasets, preprocessing, and synthetic data generation), model design (architecture, modules, and losses), and evaluation aspects (data annotation requirements and segmentation performance). We discuss these dimensions both from the viewpoint of select seminal works, and from a systematic viewpoint, examining how those choices have influenced current trends, and how their limitations should be addressed. To facilitate comparisons, we summarize all examined works in a comprehensive table as well as an interactive table available online³.

Automated Segmentation of Intracranial Thrombus on NCCT and CTA in Patients with Acute Ischemic Stroke Using a Coarse-to-Fine Deep Learning Model

BACKGROUND AND PURPOSE

Identifying the presence and extent of intracranial thrombi is crucial in selecting patients with acute ischemic stroke for treatment. This article aims to develop an automated approach to quantify thrombus on NCCT and CTA in patients with stroke.

MATERIALS AND METHODS

A total of 499 patients with large-vessel occlusion from the Safety and Efficacy of Nerinetide in Subjects Undergoing Endovascular Thrombectomy for Stroke (ESCAPE-NA1) trial were included. All patients had thin-section NCCT and CTA images. Thrombi contoured manually were used as reference standard. A deep learning approach was developed to segment thrombi automatically. Of 499 patients, 263 and 66 patients were randomly selected to train and validate the deep learning model, respectively; the remaining 170 patients were independently used for testing. The deep learning model was quantitatively compared with the reference standard using the Dice coefficient and volumetric error. The proposed deep learning model was externally tested on 83 patients with and without large-vessel occlusion from another independent trial.

RESULTS

The developed deep learning approach obtained a Dice coefficient of 70.7% (interquartile range, 58.0%-77.8%) in the internal cohort. The predicted thrombi length and volume were correlated with those of expert-contoured thrombi (r = 0.88 and 0.87, respectively; P < .001). When the derived deep learning model was applied to the external data set, the model obtained similar results in patients with large-vessel occlusion regarding the Dice coefficient (66.8%; interquartile range, 58.5%-74.6%), thrombus length (r = 0.73), and volume (r = 0.80). The model also obtained a sensitivity of 94.12% (32/34) and a specificity of 97.96% (48/49) in classifying large-vessel occlusion versus non-large-vessel occlusion.

CONCLUSIONS

The proposed deep learning method can reliably detect and measure thrombi on NCCT and CTA in patients with acute ischemic stroke.

CT medical image segmentation algorithm based on deep learning technology

For the problems of blurred edges, uneven background distribution, and many noise interferences in medical image segmentation, we proposed a medical image segmentation algorithm based on deep neural network technology, which adopts a similar U-Net backbone structure and includes two parts: encoding and decoding. Firstly, the images are passed through the encoder path with residual and convolutional structures for image feature information extraction. We added the attention mechanism module to the network jump connection to address the problems of redundant network channel dimensions and low spatial perception of complex lesions. Finally, the medical image segmentation results are obtained using the decoder path with residual and convolutional structures. To verify the validity of the model in this paper, we conducted the corresponding comparative experimental analysis, and the experimental results show that the DICE and IOU of the proposed model are 0.7826, 0.9683, 0.8904, 0.8069, and 0.9462, 0.9537 for DRIVE, ISIC2018 and COVID-19 CT datasets, respectively. The segmentation accuracy is effectively improved for medical images with complex shapes and adhesions between lesions and normal tissues.

An open-source deep learning network AVA-Net for arterial-venous area segmentation in optical coherence tomography angiography

BACKGROUND

Differential artery-vein (AV) analysis in optical coherence tomography angiography (OCTA) holds promise for the early detection of eye diseases. However, currently available methods for AV analysis are limited for binary processing of retinal vasculature in OCTA, without quantitative information of vascular perfusion intensity. This study is to develop and validate a method for quantitative AV analysis of vascular perfusion intensity.

METHOD

A deep learning network AVA-Net has been developed for automated AV area (AVA) segmentation in OCTA. Seven new OCTA features, including arterial area (AA), venous area (VA), AVA ratio (AVAR), total perfusion intensity density (T-PID), arterial PID (A-PID), venous PID (V-PID), and arterial-venous PID ratio (AV-PIDR), were extracted and tested for early detection of diabetic retinopathy (DR). Each of these seven features was evaluated for quantitative evaluation of OCTA images from healthy controls, diabetic patients without DR (NoDR), and mild DR.

RESULTS

It was observed that the area features, i.e., AA, VA and AVAR, can reveal significant differences between the control and mild DR. Vascular perfusion parameters, including T-PID and A-PID, can differentiate mild DR from control group. AV-PIDR can disclose significant differences among all three groups, i.e., control, NoDR, and mild DR. According to Bonferroni correction, the combination of A-PID and AV-PIDR can reveal significant differences in all three groups.

CONCLUSIONS

AVA-Net, which is available on GitHub for open access, enables quantitative AV analysis of AV area and vascular perfusion intensity. Comparative analysis revealed AV-PIDR as the most sensitive feature for OCTA detection of early DR. Ensemble AV feature analysis, e.g., the combination of A-PID and AV-PIDR, can further improve the performance for early DR assessment.

BBox-Guided Segmentor: Leveraging expert knowledge for accurate stroke lesion segmentation using weakly supervised bounding box prior

Stroke is one of the leading causes of death and disability in the world. Despite intensive research on automatic stroke lesion segmentation from non-invasive imaging modalities including diffusion-weighted imaging (DWI), challenges remain such as a lack of sufficient labeled data for training deep learning models and failure in detecting small lesions. In this paper, we propose BBox-Guided Segmentor, a method that significantly improves the accuracy of stroke lesion segmentation by leveraging expert knowledge. Specifically, our model uses a very coarse bounding box label provided by the expert and then performs accurate segmentation automatically. The small overhead of having the expert provide a rough bounding box leads to large performance improvement in segmentation, which is paramount to accurate stroke diagnosis. To train our model, we employ a weakly-supervised approach that uses a large number of weakly-labeled images with only bounding boxes and a small number of fully labeled images. The scarce fully labeled images are used to train a generator segmentation network, while adversarial training is used to leverage the large number of weakly-labeled images to provide additional learning signals. We evaluate our method extensively using a unique clinical dataset of 99 fully labeled cases (i.e., with full segmentation map labels) and 831 weakly labeled cases (i.e., with only bounding box labels), and the results demonstrate the superior performance of our approach over state-of-the-art stroke lesion segmentation models. We also achieve competitive performance as a SOTA fully supervised method using less than one-tenth of the complete labels. Our proposed approach has the potential to improve stroke diagnosis and treatment planning, which may lead to better patient outcomes.

Calibrating the Dice Loss to Handle Neural Network Overconfidence for Biomedical Image Segmentation

The Dice similarity coefficient (DSC) is both a widely used metric and loss function for biomedical image segmentation due to its robustness to class imbalance. However, it is well known that the DSC loss is poorly calibrated, resulting in overconfident predictions that cannot be usefully interpreted in biomedical and clinical practice. Performance is often the only metric used to evaluate segmentations produced by deep neural networks, and calibration is often neglected. However, calibration is important for translation into biomedical and clinical practice, providing crucial contextual information to model predictions for interpretation by scientists and clinicians. In this study, we provide a simple yet effective extension of the DSC loss, named the DSC++ loss, that selectively modulates the penalty associated with overconfident, incorrect predictions. As a standalone loss function, the DSC++ loss achieves significantly improved calibration over the conventional DSC loss across six well-validated open-source biomedical imaging datasets, including both 2D binary and 3D multi-class segmentation tasks. Similarly, we observe significantly improved calibration when integrating the DSC++ loss into four DSC-based loss functions. Finally, we use softmax thresholding to illustrate that well calibrated outputs enable tailoring of recall-precision bias, which is an important post-processing technique to adapt the model predictions to suit the biomedical or clinical task. The DSC++ loss overcomes the major limitation of the DSC loss, providing a suitable loss function for training deep learning segmentation models for use in biomedical and clinical practice. Source code is available at https://github.com/mlyg/DicePlusPlus .

An optimized segmentation convolutional neural network with dynamic energy loss function for 3D reconstruction of lumbar spine MR images

3D reconstruction for lumbar spine based on segmentation of Magnetic Resonance (MR) images is meaningful for diagnosis of degenerative lumbar spine diseases. However, spine MR images with unbalanced pixel distribution often cause the segmentation performance of Convolutional Neural Network (CNN) reduced. Designing a composite loss function for CNN is an effective way to enhance the segmentation capacity, yet composition loss values with fixed weight may still cause underfitting in CNN training. In this study, we designed a composite loss function with a dynamic weight, called Dynamic Energy Loss, for spine MR images segmentation. In our loss function, the weight percentage of different loss values could be dynamically adjusted during training, thus CNN could fast converge in earlier training stage and focus on detail learning in the later stage. Two datasets were used in control experiments, and the U-net CNN model with our proposed loss function achieved superior performance with Dice similarity coefficient values of 0.9484 and 0.8284 respectively, which were also verified by the Pearson correlation, Bland-Altman, and intra-class correlation coefficient analysis. Furthermore, to improve the 3D reconstruction based on the segmentation results, we proposed a filling algorithm to generate contextually related slices by computing the pixel-level difference between adjacent slices of segmented images, which could enhance the structural information of tissues between slices, and improve the performance of 3D lumbar spine model rendering. Our methods could help radiologists to build a 3D lumbar spine graphical model accurately for diagnosis while reducing burden of manual image reading.

U-Net Architecture for Prostate Segmentation: The Impact of Loss Function on System Performance

Segmentation of the prostate gland from magnetic resonance images is rapidly becoming a standard of care in prostate cancer radiotherapy treatment planning. Automating this process has the potential to improve accuracy and efficiency. However, the performance and accuracy of deep learning models varies depending on the design and optimal tuning of the hyper-parameters. In this study, we examine the effect of loss functions on the performance of deep-learning-based prostate segmentation models. A U-Net model for prostate segmentation using T2-weighted images from a local dataset was trained and performance compared when using nine different loss functions, including: Binary Cross-Entropy (BCE), Intersection over Union (IoU), Dice, BCE and Dice (BCE + Dice), weighted BCE and Dice (W (BCE + Dice)), Focal, Tversky, Focal Tversky, and Surface loss functions. Model outputs were compared using several metrics on a five-fold cross-validation set. Ranking of model performance was found to be dependent on the metric used to measure performance, but in general, W (BCE + Dice) and Focal Tversky performed well for all metrics (whole gland Dice similarity coefficient (DSC): 0.71 and 0.74; 95HD: 6.66 and 7.42; Ravid 0.05 and 0.18, respectively) and Surface loss generally ranked lowest (DSC: 0.40; 95HD: 13.64; Ravid −0.09). When comparing the performance of the models for the mid-gland, apex, and base parts of the prostate gland, the models’ performance was lower for the apex and base compared to the mid-gland. In conclusion, we have demonstrated that the performance of a deep learning model for prostate segmentation can be affected by choice of loss function. For prostate segmentation, it would appear that compound loss functions generally outperform singles loss functions such as Surface loss.

Morphological fingerprinting: Identifying patients with first-episode schizophrenia using auto-encoded morphological patterns

Although a large number of case-control statistical and machine learning studies have been conducted to investigate structural brain changes in schizophrenia, how best to measure and characterize structural abnormalities for use in classification algorithms remains an open question. In the current study, a convolutional 3D autoencoder specifically designed for discretized volumes was constructed and trained with segmented brains from 477 healthy individuals. A cohort containing 158 first-episode schizophrenia patients and 166 matched controls was fed into the trained autoencoder to generate auto-encoded morphological patterns. A classifier discriminating schizophrenia patients from healthy controls was built using 80% of the samples in this cohort by automated machine learning and validated on the remaining 20% of the samples, and this classifier was further validated on another independent cohort containing 77 first-episode schizophrenia patients and 58 matched controls acquired at a different resolution. This specially designed autoencoder allowed a satisfactory recovery of the input. With the same feature dimension, the classifier trained with autoencoded features outperformed the classifier trained with conventional morphological features by about 10% points, achieving 73.44% accuracy and 0.8 AUC on the internal validation set and 71.85% accuracy and 0.77 AUC on the external validation set. The use of features automatically learned from the segmented brain can better identify schizophrenia patients from healthy controls, but there is still a need for further improvements to establish a clinical diagnostic marker. However, with a limited sample size, the method proposed in the current study shed insight into the application of deep learning in psychiatric disorders.

Pairwise attention-enhanced adversarial model for automatic bone segmentation in CT images

Objective. Bone segmentation is a critical step in screw placement navigation. Although the deep learning methods have promoted the rapid development for bone segmentation, the local bone separation is still challenging due to irregular shapes and similar representational features.Approach. In this paper, we proposed the pairwise attention-enhanced adversarial model (Pair-SegAM) for automatic bone segmentation in computed tomography images, which includes the two parts of the segmentation model and discriminator. Considering that the distributions of the predictions from the segmentation model contains complicated semantics, we improve the discriminator to strengthen the awareness ability of the target region, improving the parsing of semantic information features. The Pair-SegAM has a pairwise structure, which uses two calculation mechanics to set up pairwise attention maps, then we utilize the semantic fusion to filter unstable regions. Therefore, the improved discriminator provides more refinement information to capture the bone outline, thus effectively enhancing the segmentation models for bone segmentation.Main results. To test the Pair-SegAM, we selected the two bone datasets for assessment. We evaluated our method against several bone segmentation models and latest adversarial models on the both datasets. The experimental results prove that our method not only exhibits superior bone segmentation performance, but also states effective generalization.Significance. Our method provides a more efficient segmentation of specific bones and has the potential to be extended to other semantic segmentation domains.

2.5D and 3D segmentation of brain metastases with deep learning on multinational MRI data

Introduction

Management of patients with brain metastases is often based on manual lesion detection and segmentation by an expert reader. This is a time- and labor-intensive process, and to that end, this work proposes an end-to-end deep learning segmentation network for a varying number of available MRI available sequences.

Methods

We adapt and evaluate a 2.5D and a 3D convolution neural network trained and tested on a retrospective multinational study from two independent centers, in addition, nnU-Net was adapted as a comparative benchmark. Segmentation and detection performance was evaluated by: (1) the dice similarity coefficient, (2) a per-metastases and the average detection sensitivity, and (3) the number of false positives.

Results

The 2.5D and 3D models achieved similar results, albeit the 2.5D model had better detection rate, whereas the 3D model had fewer false positive predictions, and nnU-Net had fewest false positives, but with the lowest detection rate. On MRI data from center 1, the 2.5D, 3D, and nnU-Net detected 79%, 71%, and 65% of all metastases; had an average per patient sensitivity of 0.88, 0.84, and 0.76; and had on average 6.2, 3.2, and 1.7 false positive predictions per patient, respectively. For center 2, the 2.5D, 3D, and nnU-Net detected 88%, 86%, and 78% of all metastases; had an average per patient sensitivity of 0.92, 0.91, and 0.85; and had on average 1.0, 0.4, and 0.1 false positive predictions per patient, respectively.

Discussion/Conclusion

Our results show that deep learning can yield highly accurate segmentations of brain metastases with few false positives in multinational data, but the accuracy degrades for metastases with an area smaller than 0.4 cm².

Automatic polyp image segmentation and cancer prediction based on deep learning

The similar shape and texture of colonic polyps and normal mucosal tissues lead to low accuracy of medical image segmentation algorithms. To solve these problems, we proposed a polyp image segmentation algorithm based on deep learning technology, which combines a HarDNet module, attention module, and multi-scale coding module with the U-Net network as the basic framework, including two stages of coding and decoding. In the encoder stage, HarDNet68 is used as the main backbone network to extract features using four null space convolutional pooling pyramids while improving the inference speed and computational efficiency; the attention mechanism module is added to the encoding and decoding network; then the model can learn the global and local feature information of the polyp image, thus having the ability to process information in both spatial and channel dimensions, to solve the problem of information loss in the encoding stage of the network and improving the performance of the segmentation network. Through comparative analysis with other algorithms, we can find that the network of this paper has a certain degree of improvement in segmentation accuracy and operation speed, which can effectively assist physicians in removing abnormal colorectal tissues and thus reduce the probability of polyp cancer, and improve the survival rate and quality of life of patients. Also, it has good generalization ability, which can provide technical support and prevention for colon cancer.

CACPU-Net: Channel attention U-net constrained by point features for crop type mapping

Crop type mapping is an indispensable topic in the agricultural field and plays an important role in agricultural intelligence. In crop type mapping, most studies focus on time series models. However, in our experimental area, the images of the crop harvest stage can be obtained from single temporal remote sensing images. Only using single temporal data for crop type mapping can reduce the difficulty of dataset production. In addition, the model of single temporal crop type mapping can also extract the spatial features of crops more effectively. In this work, we linked crop type mapping with 2D semantic segmentation and designed CACPU-Net based on single-source and single-temporal autumn Sentinel-2 satellite images. First, we used a shallow convolutional neural network, U-Net, and introduced channel attention mechanism to improve the model's ability to extract spectral features. Second, we presented the Dice to compute loss together with cross-entropy to mitigate the effects of crop class imbalance. In addition, we designed the CP module to additionally focus on hard-to-classify pixels. Our experiment was conducted on BeiDaHuang YouYi of Heilongjiang Province, which mainly grows rice, corn, soybean, and other economic crops. On the dataset we collected, through the 10-fold cross-validation experiment under the 8:1:1 dataset splitting scheme, our method achieved 93.74% overall accuracy, higher than state-of-the-art models. Compared with the previous model, our improved model has higher classification accuracy on the parcel boundary. This study provides an effective end-to-end method and a new research idea for crop type mapping. The code and the trained model are available on https://github.com/mooneed/CACPU-Net.

Visual ensemble selection of deep convolutional neural networks for 3D segmentation of breast tumors on dynamic contrast enhanced MRI

OBJECTIVES

To develop a visual ensemble selection of deep convolutional neural networks (CNN) for 3D segmentation of breast tumors using T1-weighted dynamic contrast-enhanced (T1-DCE) MRI.

METHODS

Multi-center 3D T1-DCE MRI (n = 141) were acquired for a cohort of patients diagnosed with locally advanced or aggressive breast cancer. Tumor lesions of 111 scans were equally divided between two radiologists and segmented for training. The additional 30 scans were segmented independently by both radiologists for testing. Three 3D U-Net models were trained using either post-contrast images or a combination of post-contrast and subtraction images fused at either the image or the feature level. Segmentation accuracy was evaluated quantitatively using the Dice similarity coefficient (DSC) and the Hausdorff distance (HD95) and scored qualitatively by a radiologist as excellent, useful, helpful, or unacceptable. Based on this score, a visual ensemble approach selecting the best segmentation among these three models was proposed.

RESULTS

The mean and standard deviation of DSC and HD95 between the two radiologists were equal to 77.8 ± 10.0% and 5.2 ± 5.9 mm. Using the visual ensemble selection, a DSC and HD95 equal to 78.1 ± 16.2% and 14.1 ± 40.8 mm was reached. The qualitative assessment was excellent (resp. excellent or useful) in 50% (resp. 77%).

CONCLUSION

Using subtraction images in addition to post-contrast images provided complementary information for 3D segmentation of breast lesions by CNN. A visual ensemble selection allowing the radiologist to select the most optimal segmentation obtained by the three 3D U-Net models achieved comparable results to inter-radiologist agreement, yielding 77% segmented volumes considered excellent or useful.

KEY POINTS

• Deep convolutional neural networks were developed using T1-weighted post-contrast and subtraction MRI to perform automated 3D segmentation of breast tumors. • A visual ensemble selection allowing the radiologist to choose the best segmentation among the three 3D U-Net models outperformed each of the three models. • The visual ensemble selection provided clinically useful segmentations in 77% of cases, potentially allowing for a valuable reduction of the manual 3D segmentation workload for the radiologist and greatly facilitating quantitative studies on non-invasive biomarker in breast MRI.