Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks

A novel transfer learning framework for non-uniform conductivity estimation with limited data in personalized brain stimulation

Objective. Personalized transcranial magnetic stimulation (TMS) requires individualized head models that incorporate non-uniform conductivity to enable target-specific stimulation. Accurately estimating non-uniform conductivity in individualized head models remains a challenge due to the difficulty of obtaining precise ground truth data. To address this issue, we have developed a novel transfer learning-based approach for automatically estimating non-uniform conductivity in a human head model with limited data.Approach. The proposed method complements the limitations of the previous conductivity network (CondNet) and improves the conductivity estimation accuracy. This method generates a segmentation model from T1- and T2-weighted magnetic resonance images, which is then used for conductivity estimation via transfer learning. To enhance the model's representation capability, a Transformer was incorporated into the segmentation model, while the conductivity estimation model was designed using a combination of Attention Gates and Residual Connections, enabling efficient learning even with a small amount of data.Main results. The proposed method was evaluated using 1494 images, demonstrating a 2.4% improvement in segmentation accuracy and a 29.1% increase in conductivity estimation accuracy compared with CondNet. Furthermore, the proposed method achieved superior conductivity estimation accuracy even with only three training cases, outperforming CondNet, which was trained on an adequate number of cases. The conductivity maps generated by the proposed method yielded better results in brain electrical field simulations than CondNet.Significance. These findings demonstrate the high utility of the proposed method in brain electrical field simulations and suggest its potential applicability to other medical image analysis tasks and simulations.

MLDA-Net: Multi-Level Deep Aggregation Network for 3D Nuclei Instance Segmentation

Segmentation of cell nuclei from three-dimensional (3D) volumetric fluorescence microscopy images is crucial for biological and clinical analyses. In recent years, convolutional neural networks have become the reliable 3D medical image segmentation standard. However, convolutional layers are limited by their finite receptive fields and weight-sharing mechanisms. Consequently, they struggle to effectively model long-range dependencies and spatial correlations, which may lead to inadequate nuclei segmentation. Moreover, the diversity in nuclear appearance and density poses additional challenges. This work proposes a lightweight multi-layer deep aggregation network, MLDA-Net, incorporating Wide Receptive Field Attention (WRFA). This module effectively simulates the large receptive field generated by self-attention in the Swin Transformer while requiring fewer model parameters. This design implements an extended global sensory field that enhances the ability to capture a wide range of spatial information. In addition, the multiple cross-attention (MCA) module in MLDA-Net enhances the output features of different resolutions from the encoder while maintaining global effectiveness. The Multi-Path Aggregation Feature Pyramid Network (MAFPN) receives multi-scale outputs from the MCA module, generating a robust hierarchical feature pyramid for the final prediction. MLDA-Net outperforms state-of-the-art networks, including 3DU-Net, nnFormer, UNETR, SwinUNETR, and 3DUXNET, on the 3D volumetric datasets NucMM and MitoEM. It achieves average performance improvements of 4% to 7% in F1 score, MIoU, and PQ metrics, thereby establishing new benchmark results.

Artery segmentation and atherosclerotic plaque quantification using AI for murine whole slide images stained with oil red O

Atherosclerosis is the leading cause of death in Western industrial nations. To study the etiology of plaque progression, atherosclerotic mouse models are widely used. Traditionally, analyzing the obtained histological whole slide images of Oil Red O-stained aortic roots required manual segmentation. To accelerate this process, an artificial intelligence-driven solution is proposed that comprises three stages: (1) defining the region of interest (ROI) of the aortic root using a YOLOv8l object detector, (2) applying supervised machine learning with ensembles of U-Net++ networks for artery segmentation using ROI masks, and (3) performing plaque segmentation within arterial walls with the unsupervised W-Net method. To establish a robust segmentation pipeline, we benchmark our methods using manually created masks ([Formula: see text] for artery segmentation, [Formula: see text] for plaque segmentation). A key finding is that an ensemble of U-Net++ networks applied on ROI masks outperformed single network architectures. Through a novel combination strategy, the ensemble output can be easily modified, paving the way for a quick and robust application in the lab. Finally, a case study utilizing published mouse data ([Formula: see text] slices) underscored the ability of our optimized pipeline to replicate human-made plaque predictions with a high correlation (Pearson's [Formula: see text]) and reproduce biological insights derived from manual analysis.

Computer vision-guided rapid and precise automated cranial microsurgeries in mice

A common procedure that allows interfacing with the brain is cranial microsurgery, wherein small to large craniotomies are performed on the overlying skull for insertion of neural interfaces or implantation of optically clear windows for long-term cranial observation. Performing craniotomies requires skill, time, and precision to avoid damaging the brain and dura. Here, we present a computer vision-guided craniotomy robot (CV-Craniobot) that uses machine learning to accurately estimate the dorsal skull anatomy from optical coherence tomography images. Instantaneous information of skull morphology is used by a robotic mill to rapidly and precisely remove the skull from a desired craniotomy location. We show that the CV-Craniobot can perform small (2- to 4-millimeter diameter) craniotomies with near 100% success rates within 2 minutes and large craniotomies encompassing most of the dorsal cortex in less than 10 minutes. Thus, the CV-Craniobot enables rapid and precise craniotomies, reducing surgery time compared to human practitioners and eliminating the need for long training.

Automated dentition segmentation: 3D UNet-based approach with MIScnn framework

INTRODUCTION

Advancements in technology have led to the adoption of digital workflows in dentistry, which require the segmentation of regions of interest from cone-beam computed tomography (CBCT) scans. These segmentations assist in diagnosis, treatment planning, and research. However, manual segmentation is an expensive and labor-intensive process. Therefore, automated methods, such as convolutional neural networks (CNNs), provide a more efficient way to generate segmentations from CBCT scans.

METHODS

A three-dimensional UNet-based CNN model, utilizing the Medical Image Segmentation CNN framework, was used for training and generating predictions from CBCT scans. A dataset of 351 CBCT scans, with ground-truth labels created through manual segmentation using AI-assisted segmentation software, was prepared. Data preprocessing, augmentation, and model training were performed, and the performance of the proposed CNN model was analyzed.

RESULTS

The CNN model achieved high accuracy in segmenting maxillary and mandibular teeth from CBCT scans, with average Dice Similarity Coefficient values of 91.83% and 91.35% for maxillary and mandibular teeth, respectively. Performance metrics, including Intersection over Union, precision, and recall, further confirmed the model's effectiveness.

CONCLUSIONS

The study demonstrates the efficacy of the three-dimensional UNet-based CNN model within the Medical Image Segmentation CNN framework for automated segmentation of maxillary and mandibular dentition from CBCT scans. Automated segmentation using CNNs has the potential to deliver accurate and efficient results, offering a significant advantage over traditional segmentation methods.

Uncertainty-guided pancreatic tumor auto-segmentation with Tversky ensemble

Background and purpose

Pancreatic gross tumor volume (GTV) delineation is challenging due to their variable morphology and uncertain ground truth. Previous deep learning-based auto-segmentation methods have struggled to handle tasks with uncertain ground truth and have not accommodated stylistic customizations. We aim to develop a human-in-the-loop pancreatic GTV segmentation tool using Tversky ensembles by leveraging uncertainty estimation techniques.

Material and methods

In this study, we utilized a total of 282 patients from the pancreas task of the Medical Segmentation Decathlon. Thirty patients were randomly selected to form an independent test set, while the remaining 252 patients were divided into an 80-20 % training-validation split. We incorporated Tversky loss layer during training to train a five-member segmentation ensemble with varying contouring tendencies. The Tversky ensemble predicted probability maps by estimating pixel-level segmentation uncertainties. Probability thresholding was employed on the resulting probability maps to generate the final contours, from which eleven contours were extracted for quantitative evaluation against ground truths, with variations in the threshold values.

Results

Our Tversky ensemble achieved DSC of 0.47, HD95 of 12.70 mm and MSD of 3.24 mm respectively using the optimal thresholding configuration. We outperformed the Swin-UNETR configuration that achieved the state-of-the-art result in the pancreas task of the medical segmentation decathlon.

Conclusions

Our study demonstrated the effectiveness of employing an ensemble-based uncertainty estimation technique for pancreatic tumor segmentation. The approach provided clinicians with a consensus probability map that could be fine-tuned in line with their preferences, generating contours with greater confidence.

AI-Based 3D Liver Segmentation and Volumetric Analysis in Living Donor Data

This study investigated the application of deep learning for 3-dimensional (3D) liver segmentation and volumetric analysis in living donor liver transplantation. Using abdominal computed tomography data from 55 donors, this study aimed to evaluate the liver segmentation performance of various U-Net-based models, including 3D U-Net, RU-Net, DU-Net, and RDU-Net, before and after hepatectomy. Accurate liver volume measurement is critical in liver transplantation to ensure adequate functional recovery and minimize postoperative complications. The models were trained and validated using a fivefold cross-validation approach. Performance metrics such as Dice similarity coefficient (DSC), recall, specificity, precision, and accuracy were used to assess the segmentation results. The highest segmentation accuracy was achieved in preoperative images with a DSC of 95.73 ± 1.08%, while postoperative day 7 images showed the lowest performance with a DSC of 93.14 ± 2.10%. A volumetric analysis conducted to measure hepatic resection and regeneration rates revealed an average liver resection rate of 40.52 ± 8.89% and a regeneration rate of 13.50 ± 8.95% by postoperative day 63. A regression analysis was performed on the volumetric results of the artificial intelligence model's liver resection rate and regeneration rate, and all results were statistically significant at p < 0.0001. The results indicate high reliability and clinical applicability of deep learning models in accurately measuring liver volume and assessing regenerative capacity, thus enhancing the management and recovery of liver donors.

Spatial-Temporal Information Fusion for Thyroid Nodule Segmentation in Dynamic Contrast-Enhanced MRI: A Novel Approach

This study aims to develop a novel segmentation method that utilizes spatio-temporal information for segmenting two-dimensional thyroid nodules on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). Leveraging medical morphology knowledge of the thyroid gland, we designed a semi-supervised segmentation model that first segments the thyroid gland, guiding the model to focus exclusively on the thyroid region. This approach reduces the complexity of nodule segmentation by filtering out irrelevant regions and artifacts. Then, we introduced a method to explicitly extract temporal information from DCE-MRI data and integrated this with spatial information. The fusion of spatial and temporal features enhances the model's robustness and accuracy, particularly in complex imaging scenarios. Experimental results demonstrate that the proposed method significantly improves segmentation performance across multiple state-of-the-art models. The Dice similarity coefficient (DSC) increased by 8.41%, 7.05%, 9.39%, 11.53%, 20.94%, 17.94%, and 15.65% for U-Net, U-Net +  + , SegNet, TransUnet, Swin-Unet, SSTrans-Net, and VM-Unet, respectively, and significantly improved the segmentation accuracy of nodules of different sizes. These results highlight the effectiveness of our spatial-temporal approach in achieving accurate and reliable thyroid nodule segmentation, offering a promising framework for clinical applications and future research in medical image analysis.

LiteMamba-Bound: A lightweight Mamba-based model with boundary-aware and normalized active contour loss for skin lesion segmentation

In the field of medical science, skin segmentation has gained significant importance, particularly in dermatology and skin cancer research. This domain demands high precision in distinguishing critical regions (such as lesions or moles) from healthy skin in medical images. With growing technological advancements, deep learning models have emerged as indispensable tools in addressing these challenges. One of the state-of-the-art modules revealed in recent years, the 2D Selective Scan (SS2D), based on state-space models that have already seen great success in natural language processing, has been increasingly adopted and is gradually replacing Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Leveraging the strength of this module, this paper introduces LiteMamba-Bound, a lightweight model with approximately 957K parameters, designed for skin image segmentation tasks. Notably, the Channel Attention Dual Mamba (CAD-Mamba) block is proposed within both the encoder and decoder alongside the Mix Convolution with Simple Attention bottleneck block to emphasize key features. Additionally, we propose the Reverse Attention Boundary Module to highlight challenging boundary features. Also, the Normalized Active Contour loss function presented in this paper significantly improves the model's performance compared to other loss functions. To validate performance, we conducted tests on two skin image datasets, ISIC2018 and PH2, with results consistently showing superior performance compared to other models. Our code will be made publicly available at: https://github.com/kwanghwi242/A-new-segmentation-model.

CapNet: An Automatic Attention-Based with Mixer Model for Cardiovascular Magnetic Resonance Image Segmentation

Deep neural networks have shown excellent performance in medical image segmentation, especially for cardiac images. Transformer-based models, though having advantages over convolutional neural networks due to the ability of long-range dependence learning, still have shortcomings such as having a large number of parameters and and high computational cost. Additionally, for better results, they are often pretrained on a larger data, thus requiring large memory size and increasing resource expenses. In this study, we propose a new lightweight but efficient model, namely CapNet, based on convolutions and mixing modules for cardiac segmentation from magnetic resonance images (MRI) that can be trained from scratch with a small amount of parameters. To handle varying sizes and shapes which often occur in cardiac systolic and diastolic phases, we propose attention modules for pooling, spatial, and channel information. We also propose a novel loss called the Tversky Shape Power Distance function based on the shape dissimilarity between labels and predictions that shows promising performances compared to other losses. Experiments on three public datasets including ACDC benchmark, Sunnybrook data, and MS-CMR challenge are conducted and compared with other state of the arts (SOTA). For binary segmentation, the proposed CapNet obtained the Dice similarity coefficient (DSC) of 94% and 95.93% for respectively the Endocardium and Epicardium regions with Sunnybrook dataset, 94.49% for Endocardium, and 96.82% for Epicardium with the ACDC data. Regarding the multiclass case, the average DSC by CapNet is 93.05% for the ACDC data; and the DSC scores for the MS-CMR are 94.59%, 92.22%, and 93.99% for respectively the bSSFP, T2-SPAIR, and LGE sequences of the MS-CMR. Moreover, the statistical significance analysis tests with p-value < 0.05 compared with transformer-based methods and some CNN-based approaches demonstrated that the CapNet, though having fewer training parameters, is statistically significant. The promising evaluation metrics show comparative results in both Dice and IoU indices compared to SOTA CNN-based and Transformer-based architectures.

Adaptive ensemble loss and multi-scale attention in breast ultrasound segmentation with UMA-Net

The generalization of deep learning (DL) models is critical for accurate lesion segmentation in breast ultrasound (BUS) images. Traditional DL models often struggle to generalize well due to the high frequency and scale variations inherent in BUS images. Moreover, conventional loss functions used in these models frequently result in imbalanced optimization, either prioritizing region overlap or boundary accuracy, which leads to suboptimal segmentation performance. To address these issues, we propose UMA-Net, an enhanced UNet architecture specifically designed for BUS image segmentation. UMA-Net integrates residual connections, attention mechanisms, and a bottleneck with atrous convolutions to effectively capture multi-scale contextual information without compromising spatial resolution. Additionally, we introduce an adaptive ensemble loss function that dynamically balances the contributions of different loss components during training, ensuring optimization across key segmentation metrics. This novel approach mitigates the imbalances found in conventional loss functions. We validate UMA-Net on five diverse BUS datasets-BUET, BUSI, Mendeley, OMI, and UDIAT-demonstrating superior performance. Our findings highlight the importance of addressing frequency and scale variations, confirming UMA-Net as a robust and generalizable solution for BUS image segmentation.

Skeleton-guided 3D convolutional neural network for tubular structure segmentation

PURPOSE

Accurate segmentation of tubular structures is crucial for clinical diagnosis and treatment but is challenging due to their complex branching structures and volume imbalance. The purpose of this study is to propose a 3D deep learning network that incorporates skeleton information to enhance segmentation accuracy in these tubular structures.

METHODS

Our approach employs a 3D convolutional network to extract 3D tubular structures from medical images such as CT volumetric images. We introduce a skeleton-guided module that operates on extracted features to capture and preserve the skeleton information in the segmentation results. Additionally, to effectively train our deep model in leveraging skeleton information, we propose a sigmoid-adaptive Tversky loss function which is specifically designed for skeleton segmentation.

RESULTS

We conducted experiments on two distinct 3D medical image datasets. The first dataset consisted of 90 cases of chest CT volumetric images, while the second dataset comprised 35 cases of abdominal CT volumetric images. Comparative analysis with previous segmentation approaches demonstrated the superior performance of our method. For the airway segmentation task, our method achieved an average tree length rate of 93.0%, a branch detection rate of 91.5%, and a precision rate of 90.0%. In the case of abdominal artery segmentation, our method attained an average precision rate of 97.7%, a recall rate of 91.7%, and an F-measure of 94.6%.

CONCLUSION

We present a skeleton-guided 3D convolutional network to segment tubular structures from 3D medical images. Our skeleton-guided 3D convolutional network could effectively segment small tubular structures, outperforming previous methods.

Liver tumor segmentation method combining multi-axis attention and conditional generative adversarial networks

In modern medical imaging-assisted therapies, manual annotation is commonly employed for liver and tumor segmentation in abdominal CT images. However, this approach suffers from low efficiency and poor accuracy. With the development of deep learning, automatic liver tumor segmentation algorithms based on neural networks have emerged, for the improvement of the work efficiency. However, existing liver tumor segmentation algorithms still have several limitations: (1) they often encounter the common issue of class imbalance in liver tumor segmentation tasks, where the tumor region is significantly smaller than the normal tissue region, causing models to predict more negative samples and neglect the tumor region; (2) they fail to adequately consider feature fusion between global contexts, leading to the loss of crucial information; (3) they exhibit weak perception of local details such as fuzzy boundaries, irregular shapes, and small lesions, thereby failing to capture important features. To address these issues, we propose a Multi-Axis Attention Conditional Generative Adversarial Network, referred to as MA-cGAN. Firstly, we propose the Multi-Axis attention mechanism (MA) that projects three-dimensional CT images along different axes to extract two-dimensional features. The features from different axes are then fused by using learnable factors to capture key information from different directions. Secondly, the MA is incorporated into a U-shaped segmentation network as the generator to enhance its ability to extract detailed features. Thirdly, a conditional generative adversarial network is built by combining a discriminator and a generator to enhance the stability and accuracy of the generator's segmentation results. The MA-cGAN was trained and tested on the LiTS public dataset for the liver and tumor segmentation challenge. Experimental results show that MA-cGAN improves the Dice coefficient, Hausdorff distance, average surface distance, and other metrics compared to the state-of-the-art segmentation models. The segmented liver and tumor models have clear edges, fewer false positive regions, and are closer to the true labels, which plays an active role in medical adjuvant therapy. The source code with our proposed model are available at https://github.com/jhliao0525/MA-cGAN.git.

Automated Segmentation of Lymph Nodes on Neck CT Scans Using Deep Learning

Early and accurate detection of cervical lymph nodes is essential for the optimal management and staging of patients with head and neck malignancies. Pilot studies have demonstrated the potential for radiomic and artificial intelligence (AI) approaches in increasing diagnostic accuracy for the detection and classification of lymph nodes, but implementation of many of these approaches in real-world clinical settings would necessitate an automated lymph node segmentation pipeline as a first step. In this study, we aim to develop a non-invasive deep learning (DL) algorithm for detecting and automatically segmenting cervical lymph nodes in 25,119 CT slices from 221 normal neck contrast-enhanced CT scans from patients without head and neck cancer. We focused on the most challenging task of segmentation of small lymph nodes, evaluated multiple architectures, and employed U-Net and our adapted spatial context network to detect and segment small lymph nodes measuring 5-10 mm. The developed algorithm achieved a Dice score of 0.8084, indicating its effectiveness in detecting and segmenting cervical lymph nodes despite their small size. A segmentation framework successful in this task could represent an essential initial block for future algorithms aiming to evaluate small objects such as lymph nodes in different body parts, including small lymph nodes looking normal to the naked human eye but harboring early nodal metastases.

Automatic pipeline for segmentation of LV myocardium on quantitative MR T1 maps using deep learning model and computation of radial T1 and ECV values

Native T1 mapping is a non-invasive technique used for early detection of diffused myocardial abnormalities, and it provides baseline tissue characterization. Post-contrast T1 mapping enhances tissue differentiation, enables extracellular volume (ECV) calculation, and improves myocardial viability assessment. Accurate and precise segmenting of the left ventricular (LV) myocardium on T1 maps is crucial for assessing myocardial tissue characteristics and diagnosing cardiovascular diseases (CVD). This study presents a deep learning (DL)-based pipeline for automatically segmenting LV myocardium on T1 maps and automatic computation of radial T1 and ECV values. The study employs a multicentric dataset consisting of retrospective multiparametric MRI data of 332 subjects to develop and assess the performance of the proposed method. The study compared DL architectures U-Net and Deep Res U-Net for LV myocardium segmentation, which achieved a dice similarity coefficient of 0.84 ± 0.43 and 0.85 ± 0.03, respectively. The dice similarity coefficients computed for radial sub-segmentation of the LV myocardium on basal, mid-cavity, and apical slices were 0.77 ± 0.21, 0.81 ± 0.17, and 0.61 ± 0.14, respectively. The t-test performed between ground truth vs. predicted values of native T1, post-contrast T1, and ECV showed no statistically significant difference (p > 0.05) for any of the radial sub-segments. The proposed DL method leverages the use of quantitative T1 maps for automatic LV myocardium segmentation and accurately computing radial T1 and ECV values, highlighting its potential for assisting radiologists in objective cardiac assessment and, hence, in CVD diagnostics.

Transfer Learning Approaches for Brain Metastases Screenings

BACKGROUND

In this study, we examined the effectiveness of transfer learning in improving automatic segmentation of brain metastases on magnetic resonance imaging scans, with potential applications in preventive exams and remote diagnostics.

METHODS

We trained three deep learning models on a public dataset from the ASNR-MICCAI Brain Metastasis Challenge 2024, fine-tuned them on a small private dataset, and compared their performance to models trained from scratch.

RESULTS

Results showed that models using transfer learning performed better than scratch-trained models, though the improvement was not statistically substantial. The custom Tversky and Binary Cross-Entropy loss function helped manage class imbalance and reduce false negatives, limiting missed tumor regions. Medical experts noted that, while fine-tuned models worked well with larger, well-defined tumors, they struggled with tiny, scattered tumors in complex cases.

CONCLUSIONS

This study highlights the potential of transfer learning and tailored loss functions in medical imaging, while also pointing out the models' limitations in detecting very small tumors in challenging cases.

Automated volumetric analysis of the inner ear fluid space from hydrops magnetic resonance imaging using 3D neural networks

Due to the development of magnetic resonance (MR) imaging processing technology, image-based identification of endolymphatic hydrops (EH) has played an important role in understanding inner ear illnesses, such as Meniere's disease or fluctuating sensorineural hearing loss. We segmented the inner ear, consisting of the cochlea, vestibule, and semicircular canals, using a 3D-based deep neural network model for accurate and automated EH volume ratio calculations. We built a dataset of MR cisternography (MRC) and HYDROPS-Mi2 stacks labeled with the segmentation of the perilymph fluid space and endolymph fluid space of the inner ear to devise a 3D segmentation deep neural network model. End-to-end learning was used to segment the perilymph fluid and the endolymph fluid spaces simultaneously using aligned pair data of the MRC and HYDROPS-Mi2 stacks. Consequently, the segmentation performance of the total fluid space and endolymph fluid space had Dice similarity coefficients of 0.9574 and 0.9186, respectively. In addition, the EH volume ratio calculated by experienced otologists and the EH volume ratio value predicted by the proposed deep learning model showed high agreement according to the interclass correlation coefficient (ICC) and Bland-Altman plot analysis.

Crispr-SGRU: Prediction of CRISPR/Cas9 Off-Target Activities with Mismatches and Indels Using Stacked BiGRU

CRISPR/Cas9 is a popular genome editing technology, yet its clinical application is hindered by off-target effects. Many deep learning-based methods are available for off-target prediction. However, few can predict off-target activities with insertions or deletions (indels) between single guide RNA and DNA sequence pairs. Additionally, the analysis of off-target data is challenged due to a data imbalance issue. Moreover, the prediction accuracy and interpretability remain to be improved. Here, we introduce a deep learning-based framework, named Crispr-SGRU, to predict off-target activities with mismatches and indels. This model is based on Inception and stacked BiGRU. It adopts a dice loss function to solve the inherent imbalance issue. Experimental results show our model outperforms existing methods for off-target prediction in terms of accuracy and robustness. Finally, we study the interpretability of this model through Deep SHAP and teacher-student-based knowledge distillation, and find it can provide meaningful explanations for sequence patterns regarding off-target activity.

Active Learning for Deep Visual Tracking

Convolutional neural networks (CNNs) have been successfully applied to the single target tracking task in recent years. Generally, training a deep CNN model requires numerous labeled training samples, and the number and quality of these samples directly affect the representational capability of the trained model. However, this approach is restrictive in practice, because manually labeling such a large number of training samples is time-consuming and prohibitively expensive. In this article, we propose an active learning method for deep visual tracking, which selects and annotates the unlabeled samples to train the deep CNN model. Under the guidance of active learning, the tracker based on the trained deep CNN model can achieve competitive tracking performance while reducing the labeling cost. More specifically, to ensure the diversity of selected samples, we propose an active learning method based on multiframe collaboration to select those training samples that should be and need to be annotated. Meanwhile, considering the representativeness of these selected samples, we adopt a nearest-neighbor discrimination method based on the average nearest-neighbor distance to screen isolated samples and low-quality samples. Therefore, the training samples' subset selected based on our method requires only a given budget to maintain the diversity and representativeness of the entire sample set. Furthermore, we adopt a Tversky loss to improve the bounding box estimation of our tracker, which can ensure that the tracker achieves more accurate target states. Extensive experimental results confirm that our active-learning-based tracker (ALT) achieves competitive tracking accuracy and speed compared with state-of-the-art trackers on the seven most challenging evaluation benchmarks. Project website: https://sites.google.com/view/altrack/.

Morphology and Texture-Guided Deep Neural Network for Intracranial Aneurysm Segmentation in 3D TOF-MRA

This study concentrates on the segmentation of intracranial aneurysms, a pivotal aspect of diagnosis and treatment planning. We aim to overcome the inherent instance imbalance and morphological variability by introducing a novel morphology and texture loss reweighting approach. Our innovative method involves the incorporation of tailored weights within the loss function of deep neural networks. Specifically designed to account for aneurysm size, shape, and texture, this approach strategically guides the model to focus on capturing discriminative information from imbalanced features. The study conducted extensive experimentation utilizing ADAM and RENJI TOF-MRA datasets to validate the proposed approach. The results of our experimentation demonstrate the remarkable effectiveness of the introduced methodology in improving aneurysm segmentation accuracy. By dynamically adapting to the variances present in aneurysm features, our model showcases promising outcomes for accurate diagnostic insights. The nuanced consideration of morphological and textural nuances within the loss function proves instrumental in overcoming the challenge posed by instance imbalance. In conclusion, our study presents a nuanced solution to the intricate challenge of intracranial aneurysm segmentation. The proposed morphology and texture loss reweighting approach, with its tailored weights and dynamic adaptability, proves to be instrumental in enhancing segmentation precision. The promising outcomes from our experimentation suggest the potential for accurate diagnostic insights and informed treatment strategies, marking a significant advancement in this critical domain of medical imaging.

Learning precise segmentation of neurofibrillary tangles from rapid manual point annotations

Accumulation of abnormal tau protein into neurofibrillary tangles (NFTs) is a pathologic hallmark of Alzheimer disease (AD). Accurate detection of NFTs in tissue samples can reveal relationships with clinical, demographic, and genetic features through deep phenotyping. However, expert manual analysis is time-consuming, subject to observer variability, and cannot handle the data amounts generated by modern imaging. We present a scalable, open-source, deep-learning approach to quantify NFT burden in digital whole slide images (WSIs) of post-mortem human brain tissue. To achieve this, we developed a method to generate detailed NFT boundaries directly from single-point-per-NFT annotations. We then trained a semantic segmentation model on 45 annotated 2400μm by 1200μm regions of interest (ROIs) selected from 15 unique temporal cortex WSIs of AD cases from three institutions (University of California (UC)-Davis, UC-San Diego, and Columbia University). Segmenting NFTs at the single-pixel level, the model achieved an area under the receiver operating characteristic of 0.832 and an F1 of 0.527 (196-fold over random) on a held-out test set of 664 NFTs from 20 ROIs (7 WSIs). We compared this to deep object detection, which achieved comparable but coarser-grained performance that was 60% faster. The segmentation and object detection models correlated well with expert semi-quantitative scores at the whole-slide level (Spearman's rho ρ=0.654 (p=6.50e-5) and ρ=0.513 (p=3.18e-3), respectively). We openly release this multi-institution deep-learning pipeline to provide detailed NFT spatial distribution and morphology analysis capability at a scale otherwise infeasible by manual assessment.

Derivation, external and clinical validation of a deep learning approach for detecting intracranial hypertension

Increased intracranial pressure (ICP) ≥15 mmHg is associated with adverse neurological outcomes, but needs invasive intracranial monitoring. Using the publicly available MIMIC-III Waveform Database (2000-2013) from Boston, we developed an artificial intelligence-derived biomarker for elevated ICP (aICP) for adult patients. aICP uses routinely collected extracranial waveform data as input, reducing the need for invasive monitoring. We externally validated aICP with an independent dataset from the Mount Sinai Hospital (2020-2022) in New York City. The AUROC, accuracy, sensitivity, and specificity on the external validation dataset were 0.80 (95% CI, 0.80-0.80), 73.8% (95% CI, 72.0-75.6%), 73.5% (95% CI 72.5-74.5%), and 73.0% (95% CI, 72.0-74.0%), respectively. We also present an exploratory analysis showing aICP predictions are associated with clinical phenotypes. A ten-percentile increment was associated with brain malignancy (OR = 1.68; 95% CI, 1.09-2.60), intracerebral hemorrhage (OR = 1.18; 95% CI, 1.07-1.32), and craniotomy (OR = 1.43; 95% CI, 1.12-1.84; P < 0.05 for all).

Deep Generative Adversarial Reinforcement Learning for Semi-Supervised Segmentation of Low-Contrast and Small Objects in Medical Images

Deep reinforcement learning (DRL) has demonstrated impressive performance in medical image segmentation, particularly for low-contrast and small medical objects. However, current DRL-based segmentation methods face limitations due to the optimization of error propagation in two separate stages and the need for a significant amount of labeled data. In this paper, we propose a novel deep generative adversarial reinforcement learning (DGARL) approach that, for the first time, enables end-to-end semi-supervised medical image segmentation in the DRL domain. DGARL ingeniously establishes a pipeline that integrates DRL and generative adversarial networks (GANs) to optimize both detection and segmentation tasks holistically while mutually enhancing each other. Specifically, DGARL introduces two innovative components to facilitate this integration in semi-supervised settings. First, a task-joint GAN with two discriminators links the detection results to the GAN's segmentation performance evaluation, allowing simultaneous joint evaluation and feedback. This ensures that DRL and GAN can be directly optimized based on each other's results. Second, a bidirectional exploration DRL integrates backward exploration and forward exploration to ensure the DRL agent explores the correct direction when forward exploration is disabled due to lack of explicit rewards. This mitigates the issue of unlabeled data being unable to provide rewards and rendering DRL unexplorable. Comprehensive experiments on three generalization datasets, comprising a total of 640 patients, demonstrate that our novel DGARL achieves 85.02% Dice and improves at least 1.91% for brain tumors, achieves 73.18% Dice and improves at least 4.28% for liver tumors, and achieves 70.85% Dice and improves at least 2.73% for pancreas compared to the ten most recent advanced methods, our results attest to the superiority of DGARL. Code is available at GitHub.

Self-Adaptive Teacher-Student framework for colon polyp segmentation from unannotated private data with public annotated datasets

Colon polyps have become a focal point of research due to their heightened potential to develop into appendiceal cancer, which has the highest mortality rate globally. Although numerous colon polyp segmentation methods have been developed using public polyp datasets, they tend to underperform on private datasets due to inconsistencies in data distribution and the difficulty of fine-tuning without annotations. In this paper, we propose a Self-Adaptive Teacher-Student (SATS) framework to segment colon polyps from unannotated private data by utilizing multiple publicly annotated datasets. The SATS trains multiple teacher networks on public datasets and then generates pseudo-labels on private data to assist in training a student network. To enhance the reliability of the pseudo-labels from the teacher networks, the SATS includes a newly proposed Uncertainty and Distance Fusion (UDFusion) strategy. UDFusion dynamically adjusts the pseudo-label weights based on a novel reconstruction similarity measure, innovatively bridging the gap between private and public data distributions. To ensure accurate identification and segmentation of colon polyps, the SATS also incorporates a Granular Attention Network (GANet) architecture for both teacher and student networks. GANet first identifies polyps roughly from a global perspective by encoding long-range anatomical dependencies and then refines this identification to remove false-positive areas through multi-scale background-foreground attention. The SATS framework was validated using three public datasets and one private dataset, achieving 76.30% on IoU, 86.00% on Recall, and 7.01 pixels on HD. These results outperform the existing five methods, indicating the effectiveness of this approach for colon polyp segmentation.

Exploring Unlabeled Data in Multiple Aspects for Semi-Supervised MRI Segmentation

Background: MRI segmentation offers crucial insights for automatic analysis. Although deep learning-based segmentation methods have attained cutting-edge performance, their efficacy heavily relies on vast sets of meticulously annotated data. Methods: In this study, we propose a novel semi-supervised MRI segmentation model that is able to explore unlabeled data in multiple aspects based on various semi-supervised learning technologies. Results: We compared the performance of our proposed method with other deep learning-based methods on 2 public datasets, and the results demonstrated that we have achieved Dice scores of 90.3% and 89.4% on the LA and ACDC datasets, respectively. Conclusions: We explored the synergy of various semi-supervised learning technologies for MRI segmentation, and our investigation will inspire research that focuses on designing MRI segmentation models.

SASAN: Spectrum-Axial Spatial Approach Networks for Medical Image Segmentation

Ophthalmic diseases such as central serous chorioretinopathy (CSC) significantly impair the vision of millions of people globally. Precise segmentation of choroid and macular edema is critical for diagnosing and treating these conditions. However, existing 3D medical image segmentation methods often fall short due to the heterogeneous nature and blurry features of these conditions, compounded by medical image clarity issues and noise interference arising from equipment and environmental limitations. To address these challenges, we propose the Spectrum Analysis Synergy Axial-Spatial Network (SASAN), an approach that innovatively integrates spectrum features using the Fast Fourier Transform (FFT). SASAN incorporates two key modules: the Frequency Integrated Neural Enhancer (FINE), which mitigates noise interference, and the Axial-Spatial Elementum Multiplier (ASEM), which enhances feature extraction. Additionally, we introduce the Self-Adaptive Multi-Aspect Loss ( LSM ), which balances image regions, distribution, and boundaries, adaptively updating weights during training. We compiled and meticulously annotated the Choroid and Macular Edema OCT Mega Dataset (CMED-18k), currently the world's largest dataset of its kind. Comparative analysis against 13 baselines shows our method surpasses these benchmarks, achieving the highest Dice scores and lowest HD95 in the CMED and OIMHS datasets. Our code is publicly available at https://github.com/IMOP-lab/SASAN-Pytorch.

Mass detection in automated three dimensional breast ultrasound using cascaded convolutional neural networks

PURPOSE

Early detection of breast cancer has a significant effect on reducing its mortality rate. For this purpose, automated three-dimensional breast ultrasound (3-D ABUS) has been recently used alongside mammography. The 3-D volume produced in this imaging system includes many slices. The radiologist must review all the slices to find the mass, a time-consuming task with a high probability of mistakes. Therefore, many computer-aided detection (CADe) systems have been developed to assist radiologists in this task. In this paper, we propose a novel CADe system for mass detection in 3-D ABUS images.

METHODS

The proposed system includes two cascaded convolutional neural networks. The goal of the first network is to achieve the highest possible sensitivity, and the second network's goal is to reduce false positives while maintaining high sensitivity. In both networks, an improved version of 3-D U-Net architecture is utilized in which two types of modified Inception modules are used in the encoder section. In the second network, new attention units are also added to the skip connections that receive the results of the first network as saliency maps.

RESULTS

The system was evaluated on a dataset containing 60 3-D ABUS volumes from 43 patients and 55 masses. A sensitivity of 91.48% and a mean false positive of 8.85 per patient were achieved.

CONCLUSIONS

The suggested mass detection system is fully automatic without any user interaction. The results indicate that the sensitivity and the mean FP per patient of the CADe system outperform competing techniques.

Dual Attention-Based 3D U-Net Liver Segmentation Algorithm on CT Images

The liver is a vital organ in the human body, and CT images can intuitively display its morphology. Physicians rely on liver CT images to observe its anatomical structure and areas of pathology, providing evidence for clinical diagnosis and treatment planning. To assist physicians in making accurate judgments, artificial intelligence techniques are adopted. Addressing the limitations of existing methods in liver CT image segmentation, such as weak contextual analysis and semantic information loss, we propose a novel Dual Attention-Based 3D U-Net liver segmentation algorithm on CT images. The innovations of our approach are summarized as follows: (1) We improve the 3D U-Net network by introducing residual connections to better capture multi-scale information and alleviate semantic information loss. (2) We propose the DA-Block encoder structure to enhance feature extraction capability. (3) We introduce the CBAM module into skip connections to optimize feature transmission in the encoder, reducing semantic gaps and achieving accurate liver segmentation. To validate the effectiveness of the algorithm, experiments were conducted on the LiTS dataset. The results showed that the Dice coefficient and HD95 index for liver images were 92.56% and 28.09 mm, respectively, representing an improvement of 0.84% and a reduction of 2.45 mm compared to 3D Res-UNet.

Towards reliable healthcare Imaging: conditional contrastive generative adversarial network for handling class imbalancing in MR Images

Background

Medical imaging datasets frequently encounter a data imbalance issue, where the majority of pixels correspond to healthy regions, and the minority belong to affected regions. This uneven distribution of pixels exacerbates the challenges associated with computer-aided diagnosis. The networks trained with imbalanced data tends to exhibit bias toward majority classes, often demonstrate high precision but low sensitivity.

Method

We have designed a new network based on adversarial learning namely conditional contrastive generative adversarial network (CCGAN) to tackle the problem of class imbalancing in a highly imbalancing MRI dataset. The proposed model has three new components: (1) class-specific attention, (2) region rebalancing module (RRM) and supervised contrastive-based learning network (SCoLN). The class-specific attention focuses on more discriminative areas of the input representation, capturing more relevant features. The RRM promotes a more balanced distribution of features across various regions of the input representation, ensuring a more equitable segmentation process. The generator of the CCGAN learns pixel-level segmentation by receiving feedback from the SCoLN based on the true negative and true positive maps. This process ensures that final semantic segmentation not only addresses imbalanced data issues but also enhances classification accuracy.

Results

The proposed model has shown state-of-art-performance on five highly imbalance medical image segmentation datasets. Therefore, the suggested model holds significant potential for application in medical diagnosis, in cases characterized by highly imbalanced data distributions. The CCGAN achieved the highest scores in terms of dice similarity coefficient (DSC) on various datasets: 0.965 ± 0.012 for BUS2017, 0.896 ± 0.091 for DDTI, 0.786 ± 0.046 for LiTS MICCAI 2017, 0.712 ± 1.5 for the ATLAS dataset, and 0.877 ± 1.2 for the BRATS 2015 dataset. DeepLab-V3 follows closely, securing the second-best position with DSC scores of 0.948 ± 0.010 for BUS2017, 0.895 ± 0.014 for DDTI, 0.763 ± 0.044 for LiTS MICCAI 2017, 0.696 ± 1.1 for the ATLAS dataset, and 0.846 ± 1.4 for the BRATS 2015 dataset.

Structure Preserving Cycle-Gan for Unsupervised Medical Image Domain Adaptation

The presence of domain shift in medical imaging is a common issue, which can greatly impact the performance of segmentation models when dealing with unseen image domains. This work introduces the Structure Preserving Cycle-GAN (SP Cycle-GAN) for unsupervised domain adaptation (DA) of segmentation datasets, which promotes medical structure preservation during image translation through the enforcement of a segmentation loss term in the overall Cycle-GAN training process. We demonstrate the structure preserving capability of the SP Cycle-GAN both visually and through comparison of Dice score segmentation performance for the un-supervised domain adaptation models. The SP Cycle-GAN is able to outperform baseline approaches and standard Cycle-GAN domain adaptation for binary blood vessel segmentation in the STARE and DRIVE datasets, and multi-class Left Ventricle (LV) and Myocardium segmentation in the multi-modal MM-WHS dataset. SP Cycle-GAN achieved a state of the art Myocardium segmentation Dice score (DSC) of 0.7435 for the MR to CT MM-WHS domain adaptation problem.

Efficient In-Training Adaptive Compound Loss Function Contribution Control for Medical Image Segmentation

Image segmentation plays a crucial role in many clinical applications, including disease diagnosis and monitoring. Current state-of-the-art segmentation approaches use deep neural networks that are trained on their target tasks by minimizing a loss function. Class imbalance is one of the major challenges that these networks face, where the target object is significantly underrepresented. Compound loss functions that incorporate the binary cross-entropy (BCE) and Dice loss are among the most prominent approaches to address this issue. However, determining the contribution of each individual loss to the overall compound loss function is a tedious process. It requires hyperparameter fine-tuning and multiple iterations of training, which is highly inefficient in terms of time and energy consumption. To address this issue, we propose an approach that adaptively controls the contribution of each of these individual loss functions during training. This eliminates the need for multiple fine-tuning iterations to achieve the desired precision and recall for segmentation models.

HST-MRF: Heterogeneous Swin Transformer With Multi-Receptive Field for Medical Image Segmentation

The Transformer has been successfully used in medical image segmentation due to its excellent long-range modeling capabilities. However, patch segmentation is necessary when building a Transformer class model. This process ignores the tissue structure features within patch, resulting in the loss of shallow representation information. In this study, we propose a Heterogeneous Swin Transformer with Multi-Receptive Field (HST-MRF) model that fuses patch information from different receptive fields to solve the problem of loss of feature information caused by patch segmentation. The heterogeneous Swin Transformer (HST) is the core module, which achieves the interaction of multi-receptive field patch information through heterogeneous attention and passes it to the next stage for progressive learning, thus complementing the patch structure information. We also designed a two-stage fusion module, multimodal bilinear pooling (MBP), to assist HST in further fusing multi-receptive field information and combining low-level and high-level semantic information for accurate localization of lesion regions. In addition, we developed adaptive patch embedding (APE) and soft channel attention (SCA) modules to retain more valuable information when acquiring patch embedding and filtering channel features, respectively, thereby improving model segmentation quality. We evaluated HST-MRF on multiple datasets for polyp, skin lesion and breast ultrasound segmentation tasks. Experimental results show that our proposed method outperforms state-of-the-art models and can achieve superior performance. Furthermore, we verified the effectiveness of each module and the benefits of multi-receptive field segmentation in reducing the loss of structural information through ablation experiments and qualitative analysis.

Colony context and size-dependent compensation mechanisms give rise to variations in nuclear growth trajectories

To investigate the fundamental question of how cellular variations arise across spatiotemporal scales in a population of identical healthy cells, we focused on nuclear growth in hiPS cell colonies as a model system. We generated a 3D timelapse dataset of thousands of nuclei over multiple days, and developed open-source tools for image and data analysis and an interactive timelapse viewer for exploring quantitative features of nuclear size and shape. We performed a data-driven analysis of nuclear growth variations across timescales. We found that individual nuclear volume growth trajectories arise from short timescale variations attributable to their spatiotemporal context within the colony. We identified a strikingly time-invariant volume compensation relationship between nuclear growth duration and starting volume across the population. Notably, we discovered that inheritance plays a crucial role in determining these two key nuclear growth features while other growth features are determined by their spatiotemporal context and are not inherited.

CACSNet for automatic robust classification and segmentation of carotid artery calcification on panoramic radiographs using a cascaded deep learning network

Stroke is one of the major causes of death worldwide, and is closely associated with atherosclerosis of the carotid artery. Panoramic radiographs (PRs) are routinely used in dental practice, and can be used to visualize carotid artery calcification (CAC). The purpose of this study was to automatically and robustly classify and segment CACs with large variations in size, shape, and location, and those overlapping with anatomical structures based on deep learning analysis of PRs. We developed a cascaded deep learning network (CACSNet) consisting of classification and segmentation networks for CACs on PRs. This network was trained on ground truth data accurately determined with reference to CT images using the Tversky loss function with optimized weights by balancing between precision and recall. CACSNet with EfficientNet-B4 achieved an AUC of 0.996, accuracy of 0.985, sensitivity of 0.980, and specificity of 0.988 in classification for normal or abnormal PRs. Segmentation performances for CAC lesions were 0.595 for the Jaccard index, 0.722 for the Dice similarity coefficient, 0.749 for precision, and 0.756 for recall. Our network demonstrated superior classification performance to previous methods based on PRs, and had comparable segmentation performance to studies based on other imaging modalities. Therefore, CACSNet can be used for robust classification and segmentation of CAC lesions that are morphologically variable and overlap with surrounding structures over the entire posterior inferior region of the mandibular angle on PRs.

Towards more precise automatic analysis: a systematic review of deep learning-based multi-organ segmentation

Accurate segmentation of multiple organs in the head, neck, chest, and abdomen from medical images is an essential step in computer-aided diagnosis, surgical navigation, and radiation therapy. In the past few years, with a data-driven feature extraction approach and end-to-end training, automatic deep learning-based multi-organ segmentation methods have far outperformed traditional methods and become a new research topic. This review systematically summarizes the latest research in this field. We searched Google Scholar for papers published from January 1, 2016 to December 31, 2023, using keywords "multi-organ segmentation" and "deep learning", resulting in 327 papers. We followed the PRISMA guidelines for paper selection, and 195 studies were deemed to be within the scope of this review. We summarized the two main aspects involved in multi-organ segmentation: datasets and methods. Regarding datasets, we provided an overview of existing public datasets and conducted an in-depth analysis. Concerning methods, we categorized existing approaches into three major classes: fully supervised, weakly supervised and semi-supervised, based on whether they require complete label information. We summarized the achievements of these methods in terms of segmentation accuracy. In the discussion and conclusion section, we outlined and summarized the current trends in multi-organ segmentation.

From quantitative metrics to clinical success: assessing the utility of deep learning for tumor segmentation in breast surgery

PURPOSE

Preventing positive margins is essential for ensuring favorable patient outcomes following breast-conserving surgery (BCS). Deep learning has the potential to enable this by automatically contouring the tumor and guiding resection in real time. However, evaluation of such models with respect to pathology outcomes is necessary for their successful translation into clinical practice.

METHODS

Sixteen deep learning models based on established architectures in the literature are trained on 7318 ultrasound images from 33 patients. Models are ranked by an expert based on their contours generated from images in our test set. Generated contours from each model are also analyzed using recorded cautery trajectories of five navigated BCS cases to predict margin status. Predicted margins are compared with pathology reports.

RESULTS

The best-performing model using both quantitative evaluation and our visual ranking framework achieved a mean Dice score of 0.959. Quantitative metrics are positively associated with expert visual rankings. However, the predictive value of generated contours was limited with a sensitivity of 0.750 and a specificity of 0.433 when tested against pathology reports.

CONCLUSION

We present a clinical evaluation of deep learning models trained for intraoperative tumor segmentation in breast-conserving surgery. We demonstrate that automatic contouring is limited in predicting pathology margins despite achieving high performance on quantitative metrics.

VesselBoost: A Python Toolbox for Small Blood Vessel Segmentation in Human Magnetic Resonance Angiography Data

Magnetic resonance angiography (MRA) performed at ultra-high magnetic field provides a unique opportunity to study the arteries of the living human brain at the mesoscopic level. From this, we can gain new insights into the brain's blood supply and vascular disease affecting small vessels. However, for quantitative characterization and precise representation of human angioarchitecture to, for example, inform blood-flow simulations, detailed segmentations of the smallest vessels are required. Given the success of deep learning-based methods in many segmentation tasks, we here explore their application to high-resolution MRA data, and address the difficulty of obtaining large data sets of correctly and comprehensively labelled data. We introduce VesselBoost, a vessel segmentation package, which utilizes deep learning and imperfect training labels for accurate vasculature segmentation. Combined with an innovative data augmentation technique, which leverages the resemblance of vascular structures, VesselBoost enables detailed vascular segmentations.

ELRL-MD: a deep learning approach for myocarditis diagnosis using cardiac magnetic resonance images with ensemble and reinforcement learning integration

Objective.Myocarditis poses a significant health risk, often precipitated by viral infections like coronavirus disease, and can lead to fatal cardiac complications. As a less invasive alternative to the standard diagnostic practice of endomyocardial biopsy, which is highly invasive and thus limited to severe cases, cardiac magnetic resonance (CMR) imaging offers a promising solution for detecting myocardial abnormalities.Approach.This study introduces a deep model called ELRL-MD that combines ensemble learning and reinforcement learning (RL) for effective myocarditis diagnosis from CMR images. The model begins with pre-training via the artificial bee colony (ABC) algorithm to enhance the starting point for learning. An array of convolutional neural networks (CNNs) then works in concert to extract and integrate features from CMR images for accurate diagnosis. Leveraging the Z-Alizadeh Sani myocarditis CMR dataset, the model employs RL to navigate the dataset's imbalance by conceptualizing diagnosis as a decision-making process.Main results.ELRL-DM demonstrates remarkable efficacy, surpassing other deep learning, conventional machine learning, and transfer learning models, achieving an F-measure of 88.2% and a geometric mean of 90.6%. Extensive experimentation helped pinpoint the optimal reward function settings and the perfect count of CNNs.Significance.The study addresses the primary technical challenge of inherent data imbalance in CMR imaging datasets and the risk of models converging on local optima due to suboptimal initial weight settings. Further analysis, leaving out ABC and RL components, confirmed their contributions to the model's overall performance, underscoring the effectiveness of addressing these critical technical challenges.

A medical image segmentation method for rectal tumors based on multi-scale feature retention and multiple attention mechanisms

BACKGROUND

With the continuous development of deep learning algorithms in the field of medical images, models for medical image processing based on convolutional neural networks have made great progress. Since medical images of rectal tumors are characterized by specific morphological features and complex edges that differ from natural images, achieving good segmentation results often requires a higher level of enrichment through the utilization of semantic features.

PURPOSE

The efficiency of feature extraction and utilization has been improved to some extent through enhanced hardware arithmetic and deeper networks in most models. However, problems still exist with detail loss and difficulty in feature extraction, arising from the extraction of high-level semantic features in deep networks.

METHODS

In this work, a novel medical image segmentation model has been proposed for Magnetic Resonance Imaging (MRI) image segmentation of rectal tumors. The model constructs a backbone architecture based on the idea of jump-connected feature fusion and solves the problems of detail feature loss and low segmentation accuracy using three novel modules: Multi-scale Feature Retention (MFR), Multi-branch Cross-channel Attention (MCA), and Coordinate Attention (CA).

RESULTS

Compared with existing methods, our proposed model is able to segment the tumor region more effectively, achieving 97.4% and 94.9% in Dice and mIoU metrics, respectively, exhibiting excellent segmentation performance and computational speed.

CONCLUSIONS

Our proposed model has improved the accuracy of both lesion region and tumor edge segmentation. In particular, the determination of the lesion region can help doctors identify the tumor location in clinical diagnosis, and the accurate segmentation of the tumor edge can assist doctors in judging the necessity and feasibility of surgery.

LST-AI: A deep learning ensemble for accurate MS lesion segmentation

Automated segmentation of brain white matter lesions is crucial for both clinical assessment and scientific research in multiple sclerosis (MS). Over a decade ago, we introduced an engineered lesion segmentation tool, LST. While recent lesion segmentation approaches have leveraged artificial intelligence (AI), they often remain proprietary and difficult to adopt. As an open-source tool, we present LST-AI, an advanced deep learning-based extension of LST that consists of an ensemble of three 3D U-Nets. LST-AI explicitly addresses the imbalance between white matter (WM) lesions and non-lesioned WM. It employs a composite loss function incorporating binary cross-entropy and Tversky loss to improve segmentation of the highly heterogeneous MS lesions. We train the network ensemble on 491 MS pairs of T1-weighted and FLAIR images, collected in-house from a 3T MRI scanner, and expert neuroradiologists manually segmented the utilized lesion maps for training. LST-AI also includes a lesion location annotation tool, labeling lesions as periventricular, infratentorial, and juxtacortical according to the 2017 McDonald criteria, and, additionally, as subcortical. We conduct evaluations on 103 test cases consisting of publicly available data using the Anima segmentation validation tools and compare LST-AI with several publicly available lesion segmentation models. Our empirical analysis shows that LST-AI achieves superior performance compared to existing methods. Its Dice and F1 scores exceeded 0.62, outperforming LST, SAMSEG (Sequence Adaptive Multimodal SEGmentation), and the popular nnUNet framework, which all scored below 0.56. Notably, LST-AI demonstrated exceptional performance on the MSSEG-1 challenge dataset, an international WM lesion segmentation challenge, with a Dice score of 0.65 and an F1 score of 0.63-surpassing all other competing models at the time of the challenge. With increasing lesion volume, the lesion detection rate rapidly increased with a detection rate of >75% for lesions with a volume between 10 mm3 and 100 mm3. Given its higher segmentation performance, we recommend that research groups currently using LST transition to LST-AI. To facilitate broad adoption, we are releasing LST-AI as an open-source model, available as a command-line tool, dockerized container, or Python script, enabling diverse applications across multiple platforms.

Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis

The full acceptance of Deep Learning (DL) models in the clinical field is rather low with respect to the quantity of high-performing solutions reported in the literature. End users are particularly reluctant to rely on the opaque predictions of DL models. Uncertainty quantification methods have been proposed in the literature as a potential solution, to reduce the black-box effect of DL models and increase the interpretability and the acceptability of the result by the final user. In this review, we propose an overview of the existing methods to quantify uncertainty associated with DL predictions. We focus on applications to medical image analysis, which present specific challenges due to the high dimensionality of images and their variable quality, as well as constraints associated with real-world clinical routine. Moreover, we discuss the concept of structural uncertainty, a corpus of methods to facilitate the alignment of segmentation uncertainty estimates with clinical attention. We then discuss the evaluation protocols to validate the relevance of uncertainty estimates. Finally, we highlight the open challenges for uncertainty quantification in the medical field.

Semi-Supervised Medical Image Segmentation Using Cross-Style Consistency With Shape-Aware and Local Context Constraints

Despite the remarkable progress in semi-supervised medical image segmentation methods based on deep learning, their application to real-life clinical scenarios still faces considerable challenges. For example, insufficient labeled data often makes it difficult for networks to capture the complexity and variability of the anatomical regions to be segmented. To address these problems, we design a new semi-supervised segmentation framework that aspires to produce anatomically plausible predictions. Our framework comprises two parallel networks: shape-agnostic and shape-aware networks. These networks learn from each other, enabling effective utilization of unlabeled data. Our shape-aware network implicitly introduces shape guidance to capture shape fine-grained information. Meanwhile, shape-agnostic networks employ uncertainty estimation to further obtain reliable pseudo-labels for the counterpart. We also employ a cross-style consistency strategy to enhance the network's utilization of unlabeled data. It enriches the dataset to prevent overfitting and further eases the coupling of the two networks that learn from each other. Our proposed architecture also incorporates a novel loss term that facilitates the learning of the local context of segmentation by the network, thereby enhancing the overall accuracy of prediction. Experiments on three different datasets of medical images show that our method outperforms many excellent semi-supervised segmentation methods and outperforms them in perceiving shape. The code can be seen at https://github.com/igip-liu/SLC-Net.

Deep Learning Powered Identification of Differentiated Early Mesoderm Cells from Pluripotent Stem Cells

Pluripotent stem cells can be differentiated into all three germ-layers including ecto-, endo-, and mesoderm in vitro. However, the early identification and rapid characterization of each germ-layer in response to chemical and physical induction of differentiation is limited. This is a long-standing issue for rapid and high-throughput screening to determine lineage specification efficiency. Here, we present deep learning (DL) methodologies for predicting and classifying early mesoderm cells differentiated from embryoid bodies (EBs) based on cellular and nuclear morphologies. Using a transgenic murine embryonic stem cell (mESC) line, namely OGTR1, we validated the upregulation of mesodermal genes (Brachyury (T): DsRed) in cells derived from EBs for the deep learning model training. Cells were classified into mesodermal and non-mesodermal (representing endo- and ectoderm) classes using a convolutional neural network (CNN) model called InceptionV3 which achieved a very high classification accuracy of 97% for phase images and 90% for nuclei images. In addition, we also performed image segmentation using an Attention U-Net CNN and obtained a mean intersection over union of 61% and 69% for phase-contrast and nuclear images, respectively. This work highlights the potential of integrating cell culture, imaging technologies, and deep learning methodologies in identifying lineage specification, thus contributing to the advancements in regenerative medicine. Collectively, our trained deep learning models can predict the mesoderm cells with high accuracy based on cellular and nuclear morphologies.

A Multi-Stage Feature Aggregation and Structure Awareness Network for Concrete Bridge Crack Detection

One of the most significant problems affecting a concrete bridge's safety is cracks. However, detecting concrete bridge cracks is still challenging due to their slender nature, low contrast, and background interference. The existing convolutional methods with square kernels struggle to capture crack features effectively, fail to perceive the long-range dependencies between crack regions, and have weak suppression ability for background noises, leading to low detection precision of bridge cracks. To address this problem, a multi-stage feature aggregation and structure awareness network (MFSA-Net) for pixel-level concrete bridge crack detection is proposed in this paper. Specifically, in the coding stage, a structure-aware convolution block is proposed by combining square convolution with strip convolution to perceive the linear structure of concrete bridge cracks. Square convolution is used to capture detailed local information. In contrast, strip convolution is employed to interact with the local features to establish the long-range dependence relationship between discrete crack regions. Unlike the self-attention mechanism, strip convolution also suppresses background interference near crack regions. Meanwhile, the feature attention fusion block is presented for fusing features from the encoder and decoder at the same stage, which can sharpen the edges of concrete bridge cracks. In order to fully utilize the shallow detail features and deep semantic features, the features from different stages are aggregated to obtain fine-grained segmentation results. The proposed MFSA-Net was trained and evaluated on the publicly available concrete bridge crack dataset and achieved average results of 73.74%, 77.04%, 75.30%, and 60.48% for precision, recall, F1 score, and IoU, respectively, on three typical sub-datasets, thus showing optimal performance in comparison with other existing methods. MFSA-Net also gained optimal performance on two publicly available concrete pavement crack datasets, thereby indicating its adaptability to crack detection across diverse scenarios.

A Multi-Task Network Based on Dual-Neck Structure for Autonomous Driving Perception

A vision-based autonomous driving perception system necessitates the accomplishment of a suite of tasks, including vehicle detection, drivable area segmentation, and lane line segmentation. In light of the limited computational resources available, multi-task learning has emerged as the preeminent methodology for crafting such systems. In this article, we introduce a highly efficient end-to-end multi-task learning model that showcases promising performance on all fronts. Our approach entails the development of a reliable feature extraction network by introducing a feature extraction module called C2SPD. Moreover, to account for the disparities among various tasks, we propose a dual-neck architecture. Finally, we present an optimized design for the decoders of each task. Our model evinces strong performance on the demanding BDD100K dataset, attaining remarkable accuracy (Acc) in vehicle detection and superior precision in drivable area segmentation (mIoU). In addition, this is the first work that can process these three visual perception tasks simultaneously in real time on an embedded device Atlas 200I A2 and maintain excellent accuracy.

Multi-scale V-net architecture with deep feature CRF layers for brain extraction

BACKGROUND

Brain extraction is a computational necessity for researchers using brain imaging data. However, the complex structure of the interfaces between the brain, meninges and human skull have not allowed a highly robust solution to emerge. While previous methods have used machine learning with structural and geometric priors in mind, with the development of Deep Learning (DL), there has been an increase in Neural Network based methods. Most proposed DL models focus on improving the training data despite the clear gap between groups in the amount and quality of accessible training data between.

METHODS

We propose an architecture we call Efficient V-net with Additional Conditional Random Field Layers (EVAC+). EVAC+ has 3 major characteristics: (1) a smart augmentation strategy that improves training efficiency, (2) a unique way of using a Conditional Random Fields Recurrent Layer that improves accuracy and (3) an additional loss function that fine-tunes the segmentation output. We compare our model to state-of-the-art non-DL and DL methods.

RESULTS

Results show that even with limited training resources, EVAC+ outperforms in most cases, achieving a high and stable Dice Coefficient and Jaccard Index along with a desirable lower Surface (Hausdorff) Distance. More importantly, our approach accurately segmented clinical and pediatric data, despite the fact that the training dataset only contains healthy adults.

CONCLUSIONS

Ultimately, our model provides a reliable way of accurately reducing segmentation errors in complex multi-tissue interfacing areas of the brain. We expect our method, which is publicly available and open-source, to be beneficial to a wide range of researchers.

Product Recognition for Unmanned Vending Machines

Recently, the emerging concept of "unmanned retail" has drawn more and more attention, and the unmanned retail based on the intelligent unmanned vending machines (UVMs) scene has great market demand. However, existing product recognition methods for intelligent UVMs cannot adapt to large-scale categories and have insufficient accuracy. In this article, we propose a method for large-scale categories product recognition based on intelligent UVMs. It can be divided into two parts: 1) first, we explore the similarities and differences between products through manifold learning, and then we build a hierarchical multigranularity label to constrain the learning of representation; and 2) second, we propose a hierarchical label object detection network, which mainly includes coarse-to-fine refine module (C2FRM) and multiple granularity hierarchical loss (MGHL), which are used to assist in capturing multigranularity features. The highlights of our method are mine potential similarity between large-scale category products and optimization through hierarchical multigranularity labels. Besides, we collected a large-scale product recognition dataset GOODS-85 based on the actual UVMs scenario. Experimental results and analysis demonstrate the effectiveness of the proposed product recognition methods.

Deep learning in MRI-guided radiation therapy: A systematic review

Recent advances in MRI-guided radiation therapy (MRgRT) and deep learning techniques encourage fully adaptive radiation therapy (ART), real-time MRI monitoring, and the MRI-only treatment planning workflow. Given the rapid growth and emergence of new state-of-the-art methods in these fields, we systematically review 197 studies written on or before December 31, 2022, and categorize the studies into the areas of image segmentation, image synthesis, radiomics, and real time MRI. Building from the underlying deep learning methods, we discuss their clinical importance and current challenges in facilitating small tumor segmentation, accurate x-ray attenuation information from MRI, tumor characterization and prognosis, and tumor motion tracking. In particular, we highlight the recent trends in deep learning such as the emergence of multi-modal, visual transformer, and diffusion models.

Wildfire spreading prediction using multimodal data and deep neural network approach

Predicting wildfire spread behavior is an extremely important task for many countries. On a small scale, it is possible to ensure constant monitoring of the natural landscape through ground means. However, on the scale of large countries, this becomes practically impossible due to remote and vast forest territories. The most promising source of data in this case that can provide global monitoring is remote sensing data. Currently, the main challenge is the development of an effective pipeline that combines geospatial data collection and the application of advanced machine learning algorithms. Most approaches focus on short-term fire spreading prediction and utilize data from unmanned aerial vehicles (UAVs) for this purpose. In this study, we address the challenge of predicting fire spread on a large scale and consider a forecasting horizon ranging from 1 to 5 days. We train a neural network model based on the MA-Net architecture to predict wildfire spread based on environmental and climate data, taking into account spatial distribution features. Estimating the importance of features is another critical issue in fire behavior prediction, so we analyze their contribution to the model's results. According to the experimental results, the most significant features are wind direction and land cover parameters. The F1-score for the predicted burned area varies from 0.64 to 0.68 depending on the day of prediction (from 1 to 5 days). The study was conducted in northern Russian regions and shows promise for further transfer and adaptation to other regions. This geospatial data-based artificial intelligence (AI) approach can be beneficial for supporting emergency systems and facilitating rapid decision-making.

Derivation, External Validation and Clinical Implications of a deep learning approach for intracranial pressure estimation using non-cranial waveform measurements

Importance

Increased intracranial pressure (ICP) is associated with adverse neurological outcomes, but needs invasive monitoring.

Objective

Development and validation of an AI approach for detecting increased ICP (aICP) using only non-invasive extracranial physiological waveform data.

Design

Retrospective diagnostic study of AI-assisted detection of increased ICP. We developed an AI model using exclusively extracranial waveforms, externally validated it and assessed associations with clinical outcomes.

Setting

MIMIC-III Waveform Database (2000-2013), a database derived from patients admitted to an ICU in an academic Boston hospital, was used for development of the aICP model, and to report association with neurologic outcomes. Data from Mount Sinai Hospital (2020-2022) in New York City was used for external validation.

Participants

Patients were included if they were older than 18 years, and were monitored with electrocardiograms, arterial blood pressure, respiratory impedance plethysmography and pulse oximetry. Patients who additionally had intracranial pressure monitoring were used for development (N=157) and external validation (N=56). Patients without intracranial monitors were used for association with outcomes (N=1694).

Exposures

Extracranial waveforms including electrocardiogram, arterial blood pressure, plethysmography and SpO2.

Main Outcomes and Measures

Intracranial pressure > 15 mmHg. Measures were Area under receiver operating characteristic curves (AUROCs), sensitivity, specificity, and accuracy at threshold of 0.5. We calculated odds ratios and p-values for phenotype association.

Results

The AUROC was 0.91 (95% CI, 0.90-0.91) on testing and 0.80 (95% CI, 0.80-0.80) on external validation. aICP had accuracy, sensitivity, and specificity of 73.8% (95% CI, 72.0%-75.6%), 99.5% (95% CI 99.3%-99.6%), and 76.9% (95% CI, 74.0-79.8%) on external validation. A ten-percentile increment was associated with stroke (OR=2.12; 95% CI, 1.27-3.13), brain malignancy (OR=1.68; 95% CI, 1.09-2.60), subdural hemorrhage (OR=1.66; 95% CI, 1.07-2.57), intracerebral hemorrhage (OR=1.18; 95% CI, 1.07-1.32), and procedures like percutaneous brain biopsy (OR=1.58; 95% CI, 1.15-2.18) and craniotomy (OR = 1.43; 95% CI, 1.12-1.84; P < 0.05 for all).

Conclusions and Relevance

aICP provides accurate, non-invasive estimation of increased ICP, and is associated with neurological outcomes and neurosurgical procedures in patients without intracranial monitoring.