Fine-Tuning U-Net for Ultrasound Image Segmentation: Different Layers, Different Outcomes

Wrist and elbow fracture detection and segmentation by artificial intelligence using point-of-care ultrasound

PURPOSE

Distal radius (wrist) and supracondylar (elbow) fractures are common in children presenting to Pediatric Emergency Departments (EDs). These fractures are treated conservatively or surgically depending on deformity severity. Radiographs are typically used for diagnosis but can increase wait times due to the need for radiation-safe rooms. Ultrasound (US) offers a radiation-free, faster alternative that can be performed at triage, but its noisy images are challenging to interpret.

METHODS

We developed an artificial intelligence (AI) technique for the automatic diagnosis of fractures at the wrist and elbow. While most AI for diagnosis focuses on classification results only, we applied a more explainable pipeline that used US bony region segmentation from a CNN as the basis of fracture detection. Our approach was validated on 3,822 wrist US images and 1487 elbow US images. We compared the fracture detection results from classification models and multi-channel segmentation models.

RESULTS

Combining the segmentation results with the original images showed superior performance in fracture detection at the individual patient level, achieving an accuracy of 0.889 and 0.750, sensitivity of 0.818 and 1.000, and specificity of 1.000 and 0.714 on the wrist and elbow dataset respectively. Besides, the multi-channel U-Net architecture effectively detected bony fracture regions in wrist US images.

CONCLUSION

These findings demonstrate that AI models can enable reliable, automatic wrist and elbow fracture detection in pediatric EDs, potentially reducing wait times and optimizing medical resource use.

Deep Learning-Based Medical Ultrasound Image and Video Segmentation Methods: Overview, Frontiers, and Challenges

The intricate imaging structures, artifacts, and noise present in ultrasound images and videos pose significant challenges for accurate segmentation. Deep learning has recently emerged as a prominent field, playing a crucial role in medical image processing. This paper reviews ultrasound image and video segmentation methods based on deep learning techniques, summarizing the latest developments in this field, such as diffusion and segment anything models as well as classical methods. These methods are classified into four main categories based on the characteristics of the segmentation methods. Each category is outlined and evaluated in the corresponding section. We provide a comprehensive overview of deep learning-based ultrasound image segmentation methods, evaluation metrics, and common ultrasound datasets, hoping to explain the advantages and disadvantages of each method, summarize its achievements, and discuss challenges and future trends.

Sonothrombolysis Using Microfluidically Produced Microbubbles in a Murine Model of Deep Vein Thrombosis

The need for safe and effective methods to manage deep vein thrombosis (DVT), given the risks associated with anticoagulants and thrombolytic agents, motivated research into innovative approaches to resolve blood clots. In response to this challenge, sonothrombolysis is being explored as a technique that combines microbubbles, ultrasound, and thrombolytic agents to facilitate the aggressive dissolution of thrombi. Prior studies have indicated that relatively large microbubbles accelerate the dissolution process, either in an in vitro or an arterial model. However, sonothrombolysis using large microbubbles must be evaluated in venous thromboembolism diseases, where blood flow velocity is not comparable. In this study, the efficacy of sonothrombolysis was validated in a murine model of pre-existing DVT. During therapy, microfluidically produced microbubbles of 18 μm diameter and recombinant tissue plasminogen activator (rt-PA) were administered through a tail vein catheter for 30 min, while ultrasound was applied to the abdominal region of the mice. Three-dimensional ultrasound scans were performed before and after therapy for quantification. The residual volume of the thrombi was 20% in animals post sonothrombolysis versus 52% without therapy ( p = 0.012 < 0.05 ), indicating a significant reduction in DVT volume. Histological analysis of tissue sections confirmed a reduction in DVT volume post-therapy. Therefore, large microbubbles generated from a microfluidic device show promise in ultrasound-assisted therapy to address concerns related to venous thromboembolism.

Automated Field of Interest Determination for Quantitative Ultrasound Analyses of Cervical Tissues: Toward Real-time Clinical Translation in Spontaneous Preterm Birth Risk Assessment

OBJECTIVE

Quantitative ultrasound (QUS) analysis of the human cervix is valuable for predicting spontaneous preterm birth risk. However, this approach currently requires an offline processing step wherein a medically trained analyst manually draws a free-hand field of interest (Manual FOI) for QUS computation. This offline step hinders the clinical adoption of QUS. To address this challenge, we developed a method to determine automatically the cervical FOI (Auto FOI). This study's objective is to evaluate the agreement between QUS results obtained from the Auto and Manual FOIs and assess the feasibility of using the Auto FOI to replace the Manual FOI for cervical QUS computation.

METHODS

The auto FOI method was developed and evaluated using cervical ultrasound data from 527 pregnant women, using Manual FOIs as the reference. A deep learning model was developed using the cervical B-mode image as the input to determine automatically the FOI.

RESULTS

Quantitative comparison between the Auto and Manual FOIs yielded a high pixel accuracy of 97% and a Dice coefficient of 87%. Further, the Auto FOI yielded QUS biomarker values that were highly correlated with those obtained from the Manual FOIs. For example, the Pearson correlation coefficient was 0.87 between attenuation coefficient values obtained using Auto and Manual FOIs. Further, Bland-Altman analyses showed negligible bias between QUS biomarker values computed using the Auto and Manual FOIs.

CONCLUSION

The results support the feasibility of using Auto FOIs to replace Manual FOIs in QUS computation, an important step toward the clinical adoption of QUS technology.

A comprehensive benchmarking of a U-Net based model for midbrain auto-segmentation on transcranial sonography

BACKGROUND AND OBJECTIVE

Transcranial sonography-based grading of Parkinson's Disease has gained increasing attention in recent years, and it is currently used for assistive differential diagnosis in some specialized centers. To this end, accurate midbrain segmentation is considered an important initial step. However, current practice is manual, time-consuming, and bias-prone due to the subjective nature. Relevant studies in the literature are scarce and lacks comprehensive model evaluations from application perspectives. Herein, we aimed to benchmark the best-performing U-Net model for objective, stable and robust midbrain auto-segmentation using transcranial sonography images.

METHODS

A total of 584 patients who were suspected of Parkinson's Disease were retrospectively enrolled from Beijing Tiantan Hospital. The dataset was divided into training (n = 416), validation (n = 104), and testing (n = 64) sets. Three state-of-the-art deep-learning networks (U-Net, U-Net+++, and nnU-Net) were utilized to develop segmentation models, under 5-fold cross-validation and three randomization seeds for safeguarding model validity and stability. Model evaluation was conducted in testing set in three key aspects: (i) segmentation agreement using DICE coefficients (DICE), Intersection over Union (IoU), and Hausdorff Distance (HD); (ii) model stability using standard deviations of segmentation agreement metrics; (iii) prediction time efficiency, and (iv) model robustness against various degrees of ultrasound imaging noise produced by the salt-and-pepper noise and Gaussian noise.

RESULTS

The nnU-Net achieved the best segmentation agreement (averaged DICE: 0.910, IoU: 0.836, HD: 2.793-mm) and time efficiency (1.456-s). Under mild noise corruption, the nnU-Net outperformed others with averaged scores of DICE (0.904), IoU (0.827), HD (2.941 mm) in the salt-and-pepper noise (signal-to-noise ratio, SNR = 0.95), and DICE (0.906), IoU (0.830), HD (2.967 mm) in the Gaussian noise (sigma value, σ = 0.1); by contrast, intriguingly, performance of the U-Net and U-Net+++ models were remarkably degraded. Under increasing levels of simulated noise corruption (SNR decreased from 0.95 to 0.75; σ increased from 0.1 to 0.5), the nnU-Net network exhibited marginal decline in segmentation agreement meanwhile yielding decent performance as if there were absence of noise corruption.

CONCLUSIONS

The nnU-Net model was the best-performing midbrain segmentation model in terms of segmentation agreement, stability, time efficiency and robustness, providing the community with an objective, effective and automated alternative. Moving forward, a multi-center multi-vendor study is warranted when it comes to clinical implementation.

Cross-Anatomy Transfer Learning via Shape-Aware Adaptive Fine-Tuning for 3D Vessel Segmentation

Deep learning methods have recently achieved remarkable performance in vessel segmentation applications, yet require numerous labor-intensive labeled data. To alleviate the requirement of manual annotation, transfer learning methods can potentially be used to acquire the related knowledge of tubular structures from public large-scale labeled vessel datasets for target vessel segmentation in other anatomic sites of the human body. However, the cross-anatomy domain shift is a challenging task due to the formidable discrepancy among various vessel structures in different anatomies, resulting in the limited performance of transfer learning. Therefore, we propose a cross-anatomy transfer learning framework for 3D vessel segmentation, which first generates a pre-trained model on a public hepatic vessel dataset and then adaptively fine-tunes our target segmentation network initialized from the model for segmentation of other anatomic vessels. In the framework, the adaptive fine-tuning strategy is presented to dynamically decide on the frozen or fine-tuned filters of the target network for each input sample with a proxy network. Moreover, we develop a Gaussian-based signed distance map that explicitly encodes vessel-specific shape context. The prediction of the map is added as an auxiliary task in the segmentation network to capture geometry-aware knowledge in the fine-tuning. We demonstrate the effectiveness of our method through extensive experiments on two small-scale datasets of coronary artery and brain vessel. The results indicate the proposed method effectively overcomes the discrepancy of cross-anatomy domain shift to achieve accurate vessel segmentation for these two datasets.

Cross-modal Transfer Learning Based on an Improved CycleGAN Model for Accurate Kidney Segmentation in Ultrasound Images

OBJECTIVE

Deep-learning algorithms have been widely applied in the field of automatic kidney ultrasound (US) image segmentation. However, obtaining a large number of accurate kidney labels clinically is very difficult and time-consuming. To solve this problem, we have proposed an efficient cross-modal transfer learning method to improve the performance of the segmentation network on a limited labeled kidney US dataset.

METHODS

We aim to implement an improved image-to-image translation network called Seg-CycleGAN to generate accurate annotated kidney US data from labeled abdomen computed tomography images. The Seg-CycleGAN framework primarily consists of two structures: (i) a standard CycleGAN network to visually simulate kidney US from a publicly available labeled abdomen computed tomography dataset; (ii) and a segmentation network to ensure accurate kidney anatomical structures in US images. Based on the large number of simulated kidney US images and small number of real annotated kidney US images, we then aimed to employ a fine-tuning strategy to obtain better segmentation results.

RESULTS

To validate the effectiveness of the proposed method, we tested this method on both normal and abnormal kidney US images. The experimental results showed that the proposed method achieved a segmentation accuracy of 0.8548 in dice similarity coefficient on all testing datasets and 0.7622 on the abnormal testing dataset.

CONCLUSIONS

Compared with existing data augmentation and transfer learning methods, the proposed method improved the accuracy and generalization of the kidney US image segmentation network on a limited number of training datasets. It therefore has the potential to significantly reduce annotation costs in clinical settings.

Generalizability of deep learning in organ-at-risk segmentation: A transfer learning study in cervical brachytherapy

PURPOSE

Deep learning can automate delineation in radiation therapy, reducing time and variability. Yet, its efficacy varies across different institutions, scanners, or settings, emphasizing the need for adaptable and robust models in clinical environments. Our study demonstrates the effectiveness of the transfer learning (TL) approach in enhancing the generalizability of deep learning models for auto-segmentation of organs-at-risk (OARs) in cervical brachytherapy.

METHODS

A pre-trained model was developed using 120 scans with ring and tandem applicator on a 3T magnetic resonance (MR) scanner (RT3). Four OARs were segmented and evaluated. Segmentation performance was evaluated by Volumetric Dice Similarity Coefficient (vDSC), 95 % Hausdorff Distance (HD95), surface DSC, and Added Path Length (APL). The model was fine-tuned on three out-of-distribution target groups. Pre- and post-TL outcomes, and influence of number of fine-tuning scans, were compared. A model trained with one group (Single) and a model trained with all four groups (Mixed) were evaluated on both seen and unseen data distributions.

RESULTS

TL enhanced segmentation accuracy across target groups, matching the pre-trained model's performance. The first five fine-tuning scans led to the most noticeable improvements, with performance plateauing with more data. TL outperformed training-from-scratch given the same training data. The Mixed model performed similarly to the Single model on RT3 scans but demonstrated superior performance on unseen data.

CONCLUSIONS

TL can improve a model's generalizability for OAR segmentation in MR-guided cervical brachytherapy, requiring less fine-tuning data and reduced training time. These results provide a foundation for developing adaptable models to accommodate clinical settings.

RTSeg-net: A lightweight network for real-time segmentation of fetal head and pubic symphysis from intrapartum ultrasound images

The segmentation of the fetal head (FH) and pubic symphysis (PS) from intrapartum ultrasound images plays a pivotal role in monitoring labor progression and informing crucial clinical decisions. Achieving real-time segmentation with high accuracy on systems with limited hardware capabilities presents significant challenges. To address these challenges, we propose the real-time segmentation network (RTSeg-Net), a groundbreaking lightweight deep learning model that incorporates innovative distribution shifting convolutional blocks, tokenized multilayer perceptron blocks, and efficient feature fusion blocks. Designed for optimal computational efficiency, RTSeg-Net minimizes resource demand while significantly enhancing segmentation performance. Our comprehensive evaluation on two distinct intrapartum ultrasound image datasets reveals that RTSeg-Net achieves segmentation accuracy on par with more complex state-of-the-art networks, utilizing merely 1.86 M parameters-just 6 % of their hyperparameters-and operating seven times faster, achieving a remarkable rate of 31.13 frames per second on a Jetson Nano, a device known for its limited computing capacity. These achievements underscore RTSeg-Net's potential to provide accurate, real-time segmentation on low-power devices, broadening the scope for its application across various stages of labor. By facilitating real-time, accurate ultrasound image analysis on portable, low-cost devices, RTSeg-Net promises to revolutionize intrapartum monitoring, making sophisticated diagnostic tools accessible to a wider range of healthcare settings.

Machine-to-Machine Transfer Function in Deep Learning-Based Quantitative Ultrasound

A transfer function approach was recently demonstrated to mitigate data mismatches at the acquisition level for a single ultrasound scanner in deep learning (DL)-based quantitative ultrasound (QUS). As a natural progression, we further investigate the transfer function approach and introduce a machine-to-machine (M2M) transfer function, which possesses the ability to mitigate data mismatches at a machine level. This ability opens the door to unprecedented opportunities for reducing DL model development costs, enabling the combination of data from multiple sources or scanners, or facilitating the transfer of DL models between machines. We tested the proposed method utilizing a SonixOne machine and a Verasonics machine with an L9-4 array and an L11-5 array. We conducted two types of acquisitions to obtain calibration data: stable and free-hand, using two different calibration phantoms. Without the proposed method, the mean classification accuracy when applying a model on data acquired from one system to data acquired from another system was 50%, and the mean average area under the receiver operator characteristic (ROC) curve (AUC) was 0.405. With the proposed method, mean accuracy increased to 99%, and the AUC rose to the 0.999. Additional observations include the choice of the calibration phantom led to statistically significant changes in the performance of the proposed method. Moreover, robust implementation inspired by Wiener filtering provided an effective method for transferring the domain from one machine to another machine, and it can succeed using just a single calibration view. Lastly, the proposed method proved effective when a different transducer was used in the test machine.

Segmentation of bone surface from ultrasound using a lightweight network UBS-Net

Objective. Ultrasound-assisted orthopaedic navigation held promise due to its non-ionizing feature, portability, low cost, and real-time performance. To facilitate the applications, it was critical to have accurate and real-time bone surface segmentation. Nevertheless, the imaging artifacts and low signal-to-noise ratios in the tomographical B-mode ultrasound (B-US) images created substantial challenges in bone surface detection. In this study, we presented an end-to-end lightweight US bone segmentation network (UBS-Net) for bone surface detection.Approach. We presented an end-to-end lightweight UBS-Net for bone surface detection, using the U-Net structure as the base framework and a level set loss function for improved sensitivity to bone surface detectability. A dual attention (DA) mechanism was introduced at the end of the encoder, which considered both position and channel information to obtain the correlation between the position and channel dimensions of the feature map, where axial attention (AA) replaced the traditional self-attention (SA) mechanism in the position attention module for better computational efficiency. The position attention and channel attention (CA) were combined with a two-class fusion module for the DA map. The decoding module finally completed the bone surface detection.Main Results. As a result, a frame rate of 21 frames per second (fps) in detection were achieved. It outperformed the state-of-the-art method with higher segmentation accuracy (Dice similarity coefficient: 88.76% versus 87.22%) when applied the retrospective ultrasound (US) data from 11 volunteers.Significance. The proposed UBS-Net for bone surface detection in ultrasound achieved outstanding accuracy and real-time performance. The new method out-performed the state-of-the-art methods. It had potential in US-guided orthopaedic surgery applications.

Tumor Segmentation in Intraoperative Fluorescence Images Based on Transfer Learning and Convolutional Neural Networks

OBJECTIVE

To propose a transfer learning based method of tumor segmentation in intraoperative fluorescence images, which will assist surgeons to efficiently and accurately identify the boundary of tumors of interest.

METHODS

We employed transfer learning and deep convolutional neural networks (DCNNs) for tumor segmentation. Specifically, we first pre-trained four networks on the ImageNet dataset to extract low-level features. Subsequently, we fine-tuned these networks on two fluorescence image datasets (ABFM and DTHP) separately to enhance the segmentation performance of fluorescence images. Finally, we tested the trained models on the DTHL dataset. The performance of this approach was compared and evaluated against DCNNs trained end-to-end and the traditional level-set method.

RESULTS

The transfer learning-based UNet++ model achieved high segmentation accuracies of 82.17% on the ABFM dataset, 95.61% on the DTHP dataset, and 85.49% on the DTHL test set. For the DTHP dataset, the pre-trained Deeplab v3 + network performed exceptionally well, with a segmentation accuracy of 96.48%. Furthermore, all models achieved segmentation accuracies of over 90% when dealing with the DTHP dataset.

CONCLUSION

To the best of our knowledge, this study explores tumor segmentation on intraoperative fluorescent images for the first time. The results show that compared to traditional methods, deep learning has significant advantages in improving segmentation performance. Transfer learning enables deep learning models to perform better on small-sample fluorescence image data compared to end-to-end training. This discovery provides strong support for surgeons to obtain more reliable and accurate image segmentation results during surgery.

Masked Modeling-Based Ultrasound Image Classification via Self-Supervised Learning

Recently, deep learning-based methods have emerged as the preferred approach for ultrasound data analysis. However, these methods often require large-scale annotated datasets for training deep models, which are not readily available in practical scenarios. Additionally, the presence of speckle noise and other imaging artifacts can introduce numerous hard examples for ultrasound data classification. In this paper, drawing inspiration from self-supervised learning techniques, we present a pre-training method based on mask modeling specifically designed for ultrasound data. Our study investigates three different mask modeling strategies: random masking, vertical masking, and horizontal masking. By employing these strategies, our pre-training approach aims to predict the masked portion of the ultrasound images. Notably, our method does not rely on externally labeled data, allowing us to extract representative features without the need for human annotation. Consequently, we can leverage unlabeled datasets for pre-training. Furthermore, to address the challenges posed by hard samples in ultrasound data, we propose a novel hard sample mining strategy. To evaluate the effectiveness of our proposed method, we conduct experiments on two datasets. The experimental results demonstrate that our approach outperforms other state-of-the-art methods in ultrasound image classification. This indicates the superiority of our pre-training method and its ability to extract discriminative features from ultrasound data, even in the presence of hard examples.

Segmentation of Substantia Nigra in Brain Parenchyma Sonographic Images Using Deep Learning

Currently, Parkinson's Disease (PD) is diagnosed primarily based on symptoms by experts clinicians. Neuroimaging exams represent an important tool to confirm the clinical diagnosis. Among them, Brain Parenchyma Sonography (BPS) is used to evaluate the hyperechogenicity of Substantia Nigra (SN), found in more than 90% of PD patients. In this article, we exploit a new dataset of BPS images to investigate an automatic segmentation approach for SN that can increase the accuracy of the exam and its practicability in clinical routine. This study achieves state-of-the-art performance in SN segmentation of BPS images. Indeed, it is found that the modified U-Net network scores a Dice coefficient of 0.859 ± 0.037. The results presented in this study demonstrate the feasibility and usefulness of SN automatic segmentation in BPS medical images, to the point that this study can be considered as the first stage of the development of an end-to-end CAD (Computer Aided Detection) system. Furthermore, the used dataset, which will be further enriched in the future, has proven to be very effective in supporting the training of CNNs and may pave the way for future studies in the field of CAD applied to PD.

A deep sift convolutional neural networks for total brain volume estimation from 3D ultrasound images

Preterm infants are a highly vulnerable population. The total brain volume (TBV) of these infants can be accurately estimated by brain ultrasound (US) imaging which enables a longitudinal study of early brain growth during Neonatal Intensive Care (NICU) admission. Automatic estimation of TBV from 3D images increases the diagnosis speed and evades the necessity for an expert to manually segment 3D images, which is a sophisticated and time consuming task. We develop a deep-learning approach to estimate TBV from 3D ultrasound images. It benefits from deep convolutional neural networks (CNN) with dilated residual connections and an additional layer, inspired by the fuzzy c-Means (FCM), to further separate the features into different regions, i.e. sift layer. Therefore, we call this method deep-sift convolutional neural networks (DSCNN). The proposed method is validated against three state-of-the-art methods including AlexNet-3D, ResNet-3D, and VGG-3D, for TBV estimation using two datasets acquired from two different ultrasound devices. The results highlight a strong correlation between the predictions and the observed TBV values. The regression activation maps are used to interpret DSCNN, allowing TBV estimation by exploring those pixels that are more consistent and plausible from an anatomical standpoint. Therefore, it can be used for direct estimation of TBV from 3D images without needing further image segmentation.

Natural variability in bee brain size and symmetry revealed by micro-CT imaging and deep learning

Analysing large numbers of brain samples can reveal minor, but statistically and biologically relevant variations in brain morphology that provide critical insights into animal behaviour, ecology and evolution. So far, however, such analyses have required extensive manual effort, which considerably limits the scope for comparative research. Here we used micro-CT imaging and deep learning to perform automated analyses of 3D image data from 187 honey bee and bumblebee brains. We revealed strong inter-individual variations in total brain size that are consistent across colonies and species, and may underpin behavioural variability central to complex social organisations. In addition, the bumblebee dataset showed a significant level of lateralization in optic and antennal lobes, providing a potential explanation for reported variations in visual and olfactory learning. Our fast, robust and user-friendly approach holds considerable promises for carrying out large-scale quantitative neuroanatomical comparisons across a wider range of animals. Ultimately, this will help address fundamental unresolved questions related to the evolution of animal brains and cognition.

Segmentation of Arm Ultrasound Images in Breast Cancer-Related Lymphedema: A Database and Deep Learning Algorithm

OBJECTIVE

Breast cancer treatment often causes the removal of or damage to lymph nodes of the patient's lymphatic drainage system. This side effect is the origin of Breast Cancer-Related Lymphedema (BCRL), referring to a noticeable increase in excess arm volume. Ultrasound imaging is a preferred modality for the diagnosis and progression monitoring of BCRL because of its low cost, safety, and portability. As the affected and unaffected arms look similar in B-mode ultrasound images, the thickness of the skin, subcutaneous fat, and muscle have been shown to be important biomarkers for this task. The segmentation masks are also helpful in monitoring the longitudinal changes in morphology and mechanical properties of tissue layers.

METHODS

For the first time, a publicly available ultrasound dataset containing the Radio-Frequency (RF) data of 39 subjects and manual segmentation masks by two experts, are provided. Inter- and intra-observer reproducibility studies performed on the segmentation maps show a high Dice Score Coefficient (DSC) of 0.94±0.08 and 0.92±0.06, respectively. Gated Shape Convolutional Neural Network (GSCNN) is modified for precise automatic segmentation of tissue layers, and its generalization performance is improved by the CutMix augmentation strategy.

RESULTS

We got an average DSC of 0.87±0.11 on the test set, which confirms the high performance of the method.

CONCLUSION

Automatic segmentation can pave the way for convenient and accessible staging of BCRL, and our dataset can facilitate development and validation of those methods.

SIGNIFICANCE

Timely diagnosis and treatment of BCRL have crucial importance in preventing irreversible damage.

Intelligent contour extraction approach for accurate segmentation of medical ultrasound images

Introduction: Accurate contour extraction in ultrasound images is of great interest for image-guided organ interventions and disease diagnosis. Nevertheless, it remains a problematic issue owing to the missing or ambiguous outline between organs (i.e., prostate and kidney) and surrounding tissues, the appearance of shadow artifacts, and the large variability in the shape of organs. Methods: To address these issues, we devised a method that includes four stages. In the first stage, the data sequence is acquired using an improved adaptive selection principal curve method, in which a limited number of radiologist defined data points are adopted as the prior. The second stage then uses an enhanced quantum evolution network to help acquire the optimal neural network. The third stage involves increasing the precision of the experimental outcomes after training the neural network, while using the data sequence as the input. In the final stage, the contour is smoothed using an explicable mathematical formula explained by the model parameters of the neural network. Results: Our experiments showed that our approach outperformed other current methods, including hybrid and Transformer-based deep-learning methods, achieving an average Dice similarity coefficient, Jaccard similarity coefficient, and accuracy of 95.7 ± 2.4%, 94.6 ± 2.6%, and 95.3 ± 2.6%, respectively. Discussion: This work develops an intelligent contour extraction approach on ultrasound images. Our approach obtained more satisfactory outcome compared with recent state-of-the-art approaches . The knowledge of precise boundaries of the organ is significant for the conservation of risk structures. Our developed approach has the potential to enhance disease diagnosis and therapeutic outcomes.

Artificial Intelligence Assisted Multi-modal Photoacoustic-Ultrasound Imaging for Studying Renal Tissue Function and Hemodynamics

Combined functional-anatomic imaging modalities, which integrate the benefits of visualizing gross anatomy along with the functional or metabolic information of tissue has revolutionized the world of medical imaging. However, such existing imaging modalities are very costly. An alternative option could be a hybrid modality combining contrast-enhanced ultrasound, doppler and photoacoustic imaging. In the current study, we propose an artificial intelligence assisted multi-modal imaging platform where we have used U-net model for segmenting the anatomical features from the ultrasound images obtained from an animal model study. The neural network has performed accurately for three different cases, each with a high dice score. The model was co-validated with doppler images. Further, blood perfusion and tissue oxygenation information from the predicted anatomical structures were also studied. The present findings confirm the feasibility of using this multimodal imaging modality facilitated by artificial intelligence for better understanding of the hemodynamics of the kidney.Clinical Relevance-A multi-modal imaging technique has been proposed which would provide anatomical and functional information to the clinicians for early detection and tracking of the disease prognosis. Unlike existing imaging modalities like PET-CT (Positron Emission Tomography- Computed Tomography), the proposed modality is much more costeffective and radiation free (non-ionizing nature).

Linear fine-tuning: a linear transformation based transfer strategy for deep MRI reconstruction

Introduction

Fine-tuning (FT) is a generally adopted transfer learning method for deep learning-based magnetic resonance imaging (MRI) reconstruction. In this approach, the reconstruction model is initialized with pre-trained weights derived from a source domain with ample data and subsequently updated with limited data from the target domain. However, the direct full-weight update strategy can pose the risk of "catastrophic forgetting" and overfitting, hindering its effectiveness. The goal of this study is to develop a zero-weight update transfer strategy to preserve pre-trained generic knowledge and reduce overfitting.

Methods

Based on the commonality between the source and target domains, we assume a linear transformation relationship of the optimal model weights from the source domain to the target domain. Accordingly, we propose a novel transfer strategy, linear fine-tuning (LFT), which introduces scaling and shifting (SS) factors into the pre-trained model. In contrast to FT, LFT only updates SS factors in the transfer phase, while the pre-trained weights remain fixed.

Results

To evaluate the proposed LFT, we designed three different transfer scenarios and conducted a comparative analysis of FT, LFT, and other methods at various sampling rates and data volumes. In the transfer scenario between different contrasts, LFT outperforms typical transfer strategies at various sampling rates and considerably reduces artifacts on reconstructed images. In transfer scenarios between different slice directions or anatomical structures, LFT surpasses the FT method, particularly when the target domain contains a decreasing number of training images, with a maximum improvement of up to 2.06 dB (5.89%) in peak signal-to-noise ratio.

Discussion

The LFT strategy shows great potential to address the issues of "catastrophic forgetting" and overfitting in transfer scenarios for MRI reconstruction, while reducing the reliance on the amount of data in the target domain. Linear fine-tuning is expected to shorten the development cycle of reconstruction models for adapting complicated clinical scenarios, thereby enhancing the clinical applicability of deep MRI reconstruction.

Annotation Cost Minimization for Ultrasound Image Segmentation Using Cross-Domain Transfer Learning

Deep learning techniques can help minimize inter-physician analysis variability and the medical expert workloads, thereby enabling more accurate diagnoses. However, their implementation requires large-scale annotated dataset whose acquisition incurs heavy time and human-expertise costs. Hence, to significantly minimize the annotation cost, this study presents a novel framework that enables the deployment of deep learning methods in ultrasound (US) image segmentation requiring only very limited manually annotated samples. We propose SegMix, a fast and efficient approach that exploits a segment-paste-blend concept to generate large number of annotated samples based on a few manually acquired labels. Besides, a series of US-specific augmentation strategies built upon image enhancement algorithms are introduced to make maximum use of the available limited number of manually delineated images. The feasibility of the proposed framework is validated on the left ventricle (LV) segmentation and fetal head (FH) segmentation tasks, respectively. Experimental results demonstrate that using only 10 manually annotated images, the proposed framework can achieve a Dice and JI of 82.61% and 83.92%, and 88.42% and 89.27% for LV segmentation and FH segmentation, respectively. Compared with training using the entire training set, there is over 98% of annotation cost reduction while achieving comparable segmentation performance. This indicates that the proposed framework enables satisfactory deep leaning performance when very limited number of annotated samples is available. Therefore, we believe that it can be a reliable solution for annotation cost reduction in medical image analysis.

Ultra-Attention: Automatic Recognition of Liver Ultrasound Standard Sections Based on Visual Attention Perception Structures

Acquisition of a standard section is a prerequisite for ultrasound diagnosis. For a long time, there has been a lack of clear definitions of standard liver views because of physician experience. The accurate automated scanning of standard liver sections, however, remains one of ultrasonography medicine's most important issues. In this article, we enrich and expand the classification criteria of liver ultrasound standard sections from clinical practice and propose an Ultra-Attention structured perception strategy to automate the recognition of these sections. Inspired by the attention mechanism in natural language processing, the standard liver ultrasound views will participate in the global attention algorithm as modular local images in computer vision of ultrasound images, which will significantly amplify small features that would otherwise go unnoticed. In addition to using the dropout mechanism, we also use a Part-Transfer Learning training approach to fine-tune the model's rate of convergence to increase its robustness. The proposed Ultra-Attention model outperforms various traditional convolutional neural network-based techniques, achieving the best known performance in the field with a classification accuracy of 93.2%. As part of the feature extraction procedure, we also illustrate and compare the convolutional structure and the Ultra-Attention approach. This analysis provides a reasonable view for future research on local modular feature capture in ultrasound images. By developing a standard scan guideline for liver ultrasound-based illness diagnosis, this work will advance the research on automated disease diagnosis that is directed by standard sections of liver ultrasound.

Deep-active-learning approach towards accurate right ventricular segmentation using a two-level uncertainty estimation

The Right Ventricle (RV) is currently recognised to be a significant and important prognostic factor for various pathologies. Its assessment is made possible using Magnetic Resonance Imaging (CMRI) short-axis slices. Yet, due to the challenging issues of this cavity, radiologists still perform its delineation manually, which is tedious, laborious, and time-consuming. Therefore, to automatically tackle the RV segmentation issues, Deep-Learning (DL) techniques seem to be the axis of the most recent promising approaches. Along with its potential at dealing with shape variations, DL techniques highly requires a large number of labelled images to avoid over-fitting. Subsequently, with the produced large amounts of data in the medical industry, preparing annotated datasets manually is still time-consuming, and requires high skills to be accomplished. To benefit from a significant number of labelled and unlabelled CMRI images, a Deep-Active-Learning (DAL) approach is proposed in this paper to segment the RV. Thus, three main steps are distinguished. First, a personalised labelled dataset is gathered and augmented to allow initial learning. Secondly, a U-Net based architecture is modified towards efficient initial accuracy. Finally, a two-level uncertainty estimation technique is settled to enable the selection of complementary unlabelled data. The proposed pipeline is enhanced with a customised postprocessing step, in which epistemic uncertainty and Dense Conditional Random Fields are used. The proposed approach is tested on 791 images gathered from 32 public patients and 1230 images of 50 custom subjects. The obtained results show an increased dice coefficient from 0.86 to 0.91 with a decreased Hausdorff distance from 7.55 to 7.45.

Laser ultrasonic imaging of complex defects with full-matrix capture and deep-learning extraction

Phased array-based full-matrix ultrasonic imaging has been the golden standard for the non-destructive evaluation of critical components. However, the piezoelectric phased array cannot be applied in hazardous environments and online monitoring due to its couplant requirement. The laser ultrasonic technique can readily address these challenging tasks via fully non-contact inspection, but low detection sensitivity and complicated wave mode conversion hamper its practical applications. The laser-induced full-matrix ultrasonic imaging of complex defects was displayed in this study. Full matrix data acquisition and deep learning method were adapted to the laser ultrasonic technique to overcome the existing challenges. For proof-of-concept demonstrations, simulations and experiments were conducted on an aluminum sample with representative defects. Numerical and experimental results showed good agreement, revealing the excellent imaging performance of proposed method. Compared with the total focusing method based on ray-trace model, the deep learning method could create superior images with additional quantitative information through end-to-end networks, which use the hierarchical features and generate details from all the relevant imaging and physical characteristics information. The proposed method may help assess defect formation and development at the early stage in a hazardous environment and understand the in-situ manufacturing process due to its couplant-free nature.

A comparison of manual and automated neural architecture search for white matter tract segmentation

Segmentation of white matter tracts in diffusion magnetic resonance images is an important first step in many imaging studies of the brain in health and disease. Similar to medical image segmentation in general, a popular approach to white matter tract segmentation is to use U-Net based artificial neural network architectures. Despite many suggested improvements to the U-Net architecture in recent years, there is a lack of systematic comparison of architectural variants for white matter tract segmentation. In this paper, we evaluate multiple U-Net based architectures specifically for this purpose. We compare the results of these networks to those achieved by our own various architecture changes, as well as to new U-Net architectures designed automatically via neural architecture search (NAS). To the best of our knowledge, this is the first study to systematically compare multiple U-Net based architectures for white matter tract segmentation, and the first to use NAS. We find that the recently proposed medical imaging segmentation network UNet3+ slightly outperforms the current state of the art for white matter tract segmentation, and achieves a notably better mean Dice score for segmentation of the fornix (+ 0.01 and + 0.006 mean Dice increase for left and right fornix respectively), a tract that the current state of the art model struggles to segment. UNet3+ also outperforms the current state of the art when little training data is available. Additionally, manual architecture search found that a minor segmentation improvement is observed when an additional, deeper layer is added to the U-shape of UNet3+. However, all networks, including those designed via NAS, achieve similar results, suggesting that there may be benefit in exploring networks that deviate from the general U-Net paradigm.

Investigation of optimal convolutional neural network conditions for thyroid ultrasound image analysis

Neural network models have been used to analyze thyroid ultrasound (US) images and stratify malignancy risk of the thyroid nodules. We investigated the optimal neural network condition for thyroid US image analysis. We compared scratch and transfer learning models, performed stress tests in 10% increments, and compared the performance of three threshold values. All validation results indicated superiority of the transfer learning model over the scratch model. Stress test indicated that training the algorithm using 3902 images (70%) resulted in a performance which was similar to the full dataset (5575). Threshold 0.3 yielded high sensitivity (1% false negative) and low specificity (72% false positive), while 0.7 gave low sensitivity (22% false negative) and high specificity (23% false positive). Here we showed that transfer learning was more effective than scratch learning in terms of area under curve, sensitivity, specificity and negative/positive predictive value, that about 3900 images were minimally required to demonstrate an acceptable performance, and that algorithm performance can be customized according to the population characteristics by adjusting threshold value.

Deep learning-based classification of adequate sonographic images for self-diagnosing deep vein thrombosis

BACKGROUND

Pulmonary thromboembolism is a serious disease that often occurs in disaster victims evacuated to shelters. Deep vein thrombosis is the most common reason for pulmonary thromboembolism, and early prevention is important. Medical technicians often perform ultrasonography as part of mobile medical screenings of disaster victims but reaching all isolated and scattered shelters is difficult. Therefore, deep vein thrombosis medical screening methods that can be easily performed by anyone are needed. The purpose of this study was to develop a method to automatically identify cross-sectional images suitable for deep vein thrombosis diagnosis so disaster victims can self-assess their risk of deep vein thrombosis.

METHODS

Ultrasonographic images of the popliteal vein were acquired in 20 subjects using stationary and portable ultrasound diagnostic equipment. Images were obtained by frame split from video. Images were classified as "Satisfactory," "Moderately satisfactory," and "Unsatisfactory" according to the level of popliteal vein visualization. Fine-tuning and classification were performed using ResNet101, a deep learning model.

RESULTS

Acquiring images with portable ultrasound diagnostic equipment resulted in a classification accuracy of 0.76 and an area under the receiver operating characteristic curve of 0.89. Acquiring images with stationary ultrasound diagnostic equipment resulted in a classification accuracy of 0.73 and an area under the receiver operating characteristic curve of 0.88.

CONCLUSION

A method for automatically identifying appropriate diagnostic cross-sectional ultrasonographic images of the popliteal vein was developed. This elemental technology is sufficiently accurate to automatically self-assess the risk of deep vein thrombosis by disaster victims.

Photoacoustic image synthesis with generative adversarial networks

Photoacoustic tomography (PAT) has the potential to recover morphological and functional tissue properties with high spatial resolution. However, previous attempts to solve the optical inverse problem with supervised machine learning were hampered by the absence of labeled reference data. While this bottleneck has been tackled by simulating training data, the domain gap between real and simulated images remains an unsolved challenge. We propose a novel approach to PAT image synthesis that involves subdividing the challenge of generating plausible simulations into two disjoint problems: (1) Probabilistic generation of realistic tissue morphology, and (2) pixel-wise assignment of corresponding optical and acoustic properties. The former is achieved with Generative Adversarial Networks (GANs) trained on semantically annotated medical imaging data. According to a validation study on a downstream task our approach yields more realistic synthetic images than the traditional model-based approach and could therefore become a fundamental step for deep learning-based quantitative PAT (qPAT).

Dlung: Unsupervised Few-Shot Diffeomorphic Respiratory Motion Modeling

Lung image registration plays an important role in lung analysis applications, such as respiratory motion modeling. Unsupervised learning-based image registration methods that can compute the deformation without the requirement of supervision attract much attention. However, it is noteworthy that they have two drawbacks: they do not handle the problem of limited data and do not guarantee diffeomorphic (topology-preserving) properties, especially when large deformation exists in lung scans. In this paper, we present an unsupervised few-shot learning-based diffeomorphic lung image registration, namely Dlung. We employ fine-tuning techniques to solve the problem of limited data and apply the scaling and squaring method to accomplish the diffeomorphic registration. Furthermore, atlas-based registration on spatio-temporal (4D) images is performed and thoroughly compared with baseline methods. Dlung achieves the highest accuracy with diffeomorphic properties. It constructs accurate and fast respiratory motion models with limited data. This research extends our knowledge of respiratory motion modeling.

ESTAN: Enhanced Small Tumor-Aware Network for Breast Ultrasound Image Segmentation

Breast tumor segmentation is a critical task in computer-aided diagnosis (CAD) systems for breast cancer detection because accurate tumor size, shape, and location are important for further tumor quantification and classification. However, segmenting small tumors in ultrasound images is challenging due to the speckle noise, varying tumor shapes and sizes among patients, and the existence of tumor-like image regions. Recently, deep learning-based approaches have achieved great success in biomedical image analysis, but current state-of-the-art approaches achieve poor performance for segmenting small breast tumors. In this paper, we propose a novel deep neural network architecture, namely the Enhanced Small Tumor-Aware Network (ESTAN), to accurately and robustly segment breast tumors. The Enhanced Small Tumor-Aware Network introduces two encoders to extract and fuse image context information at different scales, and utilizes row-column-wise kernels to adapt to the breast anatomy. We compare ESTAN and nine state-of-the-art approaches using seven quantitative metrics on three public breast ultrasound datasets, i.e., BUSIS, Dataset B, and BUSI. The results demonstrate that the proposed approach achieves the best overall performance and outperforms all other approaches on small tumor segmentation. Specifically, the Dice similarity coefficient (DSC) of ESTAN on the three datasets is 0.92, 0.82, and 0.78, respectively; and the DSC of ESTAN on the three datasets of small tumors is 0.89, 0.80, and 0.81, respectively.

An efficient annotated data generation method for echocardiographic image segmentation

BACKGROUND

In recent years, deep learning techniques have demonstrated promising performances in echocardiography (echo) data segmentation, which constitutes a critical step in the diagnosis and prognosis of cardiovascular diseases (CVDs). However, their successful implementation requires large number and high-quality annotated samples, whose acquisition is arduous and expertise-demanding. To this end, this study aims at circumventing the tedious, time-consuming and expertise-demanding data annotation involved in deep learning-based echo data segmentation.

METHODS

We propose a two-phase framework for fast generation of annotated echo data needed for implementing intelligent cardiac structure segmentation systems. First, multi-size and multi-orientation cardiac structures are simulated leveraging polynomial fitting method. Second, the obtained cardiac structures are embedded onto curated endoscopic ultrasound images using Fourier Transform algorithm, resulting in pairs of annotated samples. The practical significance of the proposed framework is validated through using the generated realistic annotated images as auxiliary dataset to pretrain deep learning models for automatic segmentation of left ventricle and left ventricle wall in real echo data, respectively.

RESULTS

Extensive experimental analyses indicate that compared with training from scratch, fine-tuning after pretraining with the generated dataset always results in significant performance improvement whereby the improvement margins in terms of Dice and IoU can reach 12.9% and 7.74%, respectively.

CONCLUSION

The proposed framework has great potential to overcome the shortage of labeled data hampering the deployment of deep learning approaches in echo data analysis.

Development and inter-institutional validation of an automatic vertebral-body misalignment error detector for Cone-Beam CT guided radiotherapy

BACKGROUND

In Cone-Beam Computed Tomography (CBCT) guided radiotherapy, off-by-one vertebral body misalignments are rare but serious errors which lead to wrong-site treatments.

PURPOSE

An automatic error detection algorithm was developed that uses a three-branch convolutional neural network error detection model (EDM) to detect off-by-one vertebral body misalignments using planning computed tomography (CT) images and setup CBCT images.

METHODS

Algorithm training and test data consisted of planning CTs and CBCTs from 480 patients undergoing radiotherapy treatment in the thoracic and abdominal regions at two radiotherapy clinics. The clinically applied registration was used to derive true-negative (no error) data. The setup and planning images were then misaligned by one vertebral body in both the superior and inferior directions, simulating the most likely misalignment scenarios. For each of the aligned and misaligned 3D image pairs, 2D slice pairs were automatically extracted in each anatomical plane about a point within the vertebral column. The three slice pairs obtained were then inputted to the EDM which returned a probability of vertebral misalignment. One model (EDM₁ ) was trained solely on data from institution #1. EDM₁ was further trained using a lower learning rate on a dataset from institution #2 to produce a fine-tuned model, EDM₂ . Another model, EDM₃ , was trained from scratch using a training dataset composed of data from both institutions. These three models were validated on a randomly selected and unseen dataset composed of images from both institutions, for a total of 303 image pairs. The model performances were quantified using a receiver operating characteristic analysis. Due to the rarity of vertebral body misalignments in the clinic, a minimum threshold value yielding a specificity of at least 99% was selected. Using this threshold, the sensitivity was calculated for each model, on each institution's test set separately.

RESULTS

When applied to the combined test set, EDM₁ , EDM₂ , and EDM₃ resulted in an area under curve of 99.5%, 99.4% and 99.5%, respectively. EDM₁ achieved a sensitivity of 96% and 88% on Institution #1 and Institution #2 test set, respectively. EDM₂ obtained a sensitivity of 95% on each institution's test set. EDM₃ achieved a sensitivity of 95% and 88% on Institution #1 and Institution #2 test set, respectively.

CONCLUSION

The proposed algorithm demonstrated accuracy in identifying off-by-one vertebral body misalignments in CBCT-guided radiotherapy that was sufficiently high to allow for practical implementation. It was found that fine-tuning the model on a multi-facility dataset can further enhance the generalizability of the algorithm. This article is protected by copyright. All rights reserved.

Annotation-guided encoder-decoder network for bone extraction in ultrasound-assisted orthopedic surgery

The patients and surgeons are usually exposed in massive ionizing radiation during fluoroscopy-based navigation orthopedic surgery. Comparatively, ultrasound-assisted orthopedic surgery could not only decrease the risk of radiation but also provide rich navigation information. However, due to the artifacts in ultrasound images, the extraction of bone structure from ultrasound sequences can be a particularly difficult task, which leads to some major challenges in ultrasound-assisted orthopedic navigation. In this paper, we propose an annotation-guided encoder-decoder network (AGN) to extract bone structure from the radiation-free ultrasound sequences. Specifically, the variability of the ultrasound probe's pose leads to the change of the ultrasound frame during the acquisition of ultrasound sequences. Therefore, a feature alignment module deployed in the AGN model is used to achieve reliable matching across ultrasound frames. Moreover, inspired by the interactive ultrasound analysis, where user annotated foreground information can help target extraction, our AGN model incorporates the annotation information obtained by Siamese networks. Experimental results validated that the AGN model not only produced better bone surface extraction than state-of-the-art methods (IOU: 0.92 versus. 0.88), but also achieved almost real-time extraction with the speed about 15 frames per second. In addition, the acquired bone surface further provided radiation-free 3D intraoperative bone structure for the intuitive navigation of orthopedic surgery.

A retrospective study of 3D deep learning approach incorporating coordinate information to improve the segmentation of pre- and post-operative abdominal aortic aneurysm

Abdominal aortic aneurysm (AAA) is one of the most common diseases worldwide. 3D segmentation of AAA provides useful information for surgical decisions and follow-up treatment. However, existing segmentation methods are time consuming and not practical in routine use. In this article, the segmentation task will be addressed automatically using a deep learning based approach which has been proved to successfully solve several medical imaging problems with excellent performances. This article therefore proposes a new solution of AAA segmentation using deep learning in a type of 3D convolutional neural network (CNN) architecture that also incorporates coordinate information. The tested CNNs are UNet, AG-DSV-UNet, VNet, ResNetMed and DenseVoxNet. The 3D-CNNs are trained with a dataset of high resolution (256 × 256) non-contrast and post-contrast CT images containing 64 slices from each of 200 patients. The dataset consists of contiguous CT slices without augmentation and no post-processing step. The experiments show that incorporation of coordinate information improves the segmentation results. The best accuracies on non-contrast and contrast-enhanced images have average dice scores of 97.13% and 96.74%, respectively. Transfer learning from a pre-trained network of a pre-operative dataset to post-operative endovascular aneurysm repair (EVAR) was also performed. The segmentation accuracy of post-operative EVAR using transfer learning on non-contrast and contrast-enhanced CT datasets achieved the best dice scores of 94.90% and 95.66%, respectively.

Investigating Shift Variance of Convolutional Neural Networks in Ultrasound Image Segmentation

While accuracy is an evident criterion for ultrasound image segmentation, output consistency across different tests is equally crucial for tracking changes in regions of interest in applications such as monitoring the patients' response to treatment, measuring the progression or regression of the disease, reaching a diagnosis, or treatment planning. Convolutional neural networks (CNNs) have attracted rapidly growing interest in automatic ultrasound image segmentation recently. However, CNNs are not shift-equivariant, meaning that, if the input translates, e.g., in the lateral direction by one pixel, the output segmentation may drastically change. To the best of our knowledge, this problem has not been studied in ultrasound image segmentation or even more broadly in ultrasound images. Herein, we investigate and quantify the shift-variance problem of CNNs in this application and further evaluate the performance of a recently published technique, called BlurPooling, for addressing the problem. In addition, we propose the Pyramidal BlurPooling method that outperforms BlurPooling in both output consistency and segmentation accuracy. Finally, we demonstrate that data augmentation is not a replacement for the proposed method. Source code is available at http://code.sonography.ai.

CT2US: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data

Accurate segmentation of kidney in ultrasound images is a vital procedure in clinical diagnosis and interventional operation. In recent years, deep learning technology has demonstrated promising prospects in medical image analysis. However, due to the inherent problems of ultrasound images, data with annotations are scarce and arduous to acquire, hampering the application of data-hungry deep learning methods. In this paper, we propose cross-modal transfer learning from computerized tomography (CT) to ultrasound (US) by leveraging annotated data in the CT modality. In particular, we adopt cycle generative adversarial network (CycleGAN) to synthesize US images from CT data and construct a transition dataset to mitigate the immense domain discrepancy between US and CT. Mainstream convolutional neural networks such as U-Net, U-Res, PSPNet, and DeepLab v3+ are pretrained on the transition dataset and then transferred to real US images. We first trained CNN models on a data set composed of 50 ultrasound images and validated them on a validation set composed of 30 ultrasound images. In addition, we selected 82 ultrasound images from another hospital to construct a cross-site data set to verify the generalization performance of the models. The experimental results show that with our proposed transfer learning strategy, the segmentation accuracy in dice similarity coefficient (DSC) reaches 0.853 for U-Net, 0.850 for U-Res, 0.826 for PSPNet and 0.827 for DeepLab v3+ on the cross-site test set. Compared with training from scratch, the accuracy improvement was 0.127, 0.097, 0.105 and 0.036 respectively. Our transfer learning strategy effectively improves the accuracy and generalization ability of ultrasound image segmentation model with limited training data.

Robust Scatterer Number Density Segmentation of Ultrasound Images

Quantitative ultrasound (QUS) aims to reveal information about the tissue microstructure using backscattered echo signals from clinical scanners. Among different QUS parameters, scatterer number density is an important property that can affect the estimation of other QUS parameters. Scatterer number density can be classified into high or low scatterer densities. If there are more than ten scatterers inside the resolution cell, the envelope data are considered as fully developed speckle (FDS) and, otherwise, as underdeveloped speckle (UDS). In conventional methods, the envelope data are divided into small overlapping windows (a strategy here we refer to as patching), and statistical parameters, such as SNR and skewness, are employed to classify each patch of envelope data. However, these parameters are system-dependent, meaning that their distribution can change by the imaging settings and patch size. Therefore, reference phantoms that have known scatterer number density are imaged with the same imaging settings to mitigate system dependency. In this article, we aim to segment regions of ultrasound data without any patching. A large dataset is generated, which has different shapes of scatterer number density and mean scatterer amplitude using a fast simulation method. We employ a convolutional neural network (CNN) for the segmentation task and investigate the effect of domain shift when the network is tested on different datasets with different imaging settings. Nakagami parametric image is employed for multitask learning to improve performance. Furthermore, inspired by the reference phantom methods in QUS, a domain adaptation stage is proposed, which requires only two frames of data from FDS and UDS classes. We evaluate our method for different experimental phantoms and in vivo data.

The Accuracy and Radiomics Feature Effects of Multiple U-net-Based Automatic Segmentation Models for Transvaginal Ultrasound Images of Cervical Cancer

Ultrasound (US) imaging has been recognized and widely used as a screening and diagnostic imaging modality for cervical cancer all over the world. However, few studies have investigated the U-net-based automatic segmentation models for cervical cancer on US images and investigated the effects of automatic segmentations on radiomics features. A total of 1102 transvaginal US images from 796 cervical cancer patients were collected and randomly divided into training (800), validation (100) and test sets (202), respectively, in this study. Four U-net models (U-net, U-net with ResNet, context encoder network (CE-net), and Attention U-net) were adapted to segment the target of cervical cancer automatically on these US images. Radiomics features were extracted and evaluated from both manually and automatically segmented area. The mean Dice similarity coefficient (DSC) of U-net, Attention U-net, CE-net, and U-net with ResNet were 0.88, 0.89, 0.88, and 0.90, respectively. The average Pearson coefficients for the evaluation of the reliability of US image-based radiomics were 0.94, 0.96, 0.94, and 0.95 for U-net, U-net with ResNet, Attention U-net, and CE-net, respectively, in their comparison with manual segmentation. The reproducibility of the radiomics parameters evaluated by intraclass correlation coefficients (ICC) showed robustness of automatic segmentation with an average ICC coefficient of 0.99. In conclusion, high accuracy of U-net-based automatic segmentations was achieved in delineating the target area of cervical cancer US images. It is feasible and reliable for further radiomics studies with features extracted from automatic segmented target areas.

TypeSeg: A type-aware encoder-decoder network for multi-type ultrasound images co-segmentation

PURPOSE

As a portable and radiation-free imaging modality, ultrasound can be easily used to image various types of tissue structures. It is important to develop a method which supports the multi-type ultrasound images co-segmentation. However, state-of-the-art ultrasound segmentation methods commonly only focus on the single type images or ignore the type-aware information.

METHODS

To solve the above problem, this work proposes a novel type-aware encoder-decoder network (TypeSeg) for the multi-type ultrasound images co-segmentation. First, we develop a type-aware metric learning module to find an optimum latent feature space where the ultrasound images of the same types are close and that of the different types are separated by a certain margin. Second, depending on the extracted features, a decision module decides whether the input ultrasound images have the common tissue type or not, and the encoder-decoder network produces a segmentation mask accordingly.

RESULTS

We evaluate the performance of the proposed TypeSeg model on the ultrasound dataset that contains four types of tissues. The proposed TypeSeg model achieves the overall best results with the mean IOU score of 87.51% ± 3.93% for the multi-type ultrasound images.

CONCLUSION

The experimental results indicate that the proposed method outperforms all the compared state-of-the-art algorithms for the multi-type ultrasound images co-segmentation task.

A 3D deep learning approach to epicardial fat segmentation in non-contrast and post-contrast cardiac CT images

Epicardial fat (ECF) is localized fat surrounding the heart muscle or myocardium and enclosed by the thin-layer pericardium membrane. Segmenting the ECF is one of the most difficult medical image segmentation tasks. Since the epicardial fat is infiltrated into the groove between cardiac chambers and is contiguous with cardiac muscle, segmentation requires location and voxel intensity. Recently, deep learning methods have been effectively used to solve medical image segmentation problems in several domains with state-of-the-art performance. This paper presents a novel approach to 3D segmentation of ECF by integrating attention gates and deep supervision into the 3D U-Net deep learning architecture. The proposed method shows significant improvement of the segmentation performance, when compared with standard 3D U-Net. The experiments show excellent performance on non-contrast CT datasets with average Dice scores of 90.06%. Transfer learning from a pre-trained model of a non-contrast CT to contrast-enhanced CT dataset was also performed. The segmentation accuracy on the contrast-enhanced CT dataset achieved a Dice score of 88.16%.

A Robust Breast ultrasound segmentation method under noisy annotations

BACKGROUND AND OBJECTIVE

A large-scale training data and accurate annotations are fundamental for current segmentation networks. However, the characteristic artifacts of ultrasound images always make the annotation task complicated, such as attenuation, speckle, shadows and signal dropout. Further complications arise as the contrast between the region of interest and background is often low. Without double-check from professionals, it is hard to guarantee that there is no noisy annotation in segmentation datasets. However, among the deep learning methods applied to ultrasound segmentation so far, no one can solve this problem.

METHOD

Given a dataset with poorly labeled masks, including a certain amount of noises, we propose an end-to-end noisy annotation tolerance network (NAT-Net). NAT-Net can detect noise by the proposed noise index (NI) and dynamically correct noisy annotations in the training stage. Simultaneously, noise index is used to correct the noise along with the output of the learning model. This method does not need any auxiliary clean datasets or prior knowledge of noise distributions, so it is more general, robust and easier to apply than the existing methods.

RESULTS

NAT-Net outperforms previous state-of-the-art methods on synthesized data with different noise ratio. For real-world dataset with more complex noise types, the IoU of NAT-Net is higher than that of state-of-art approaches by nearly 6%. Experimental results show that our method can also achieve good results compared with the existing methods for clean dataset.

CONCLUSION

The NAT-Net reduces manual interaction of data annotation, reduces dependence on medical personnel. After tumor segmentation, disease diagnosis efficiency is improved, which provides an auxiliary strategies for subsequent medical diagnosis systems based on ultrasound.

A Deep Learning-Based Automatic First-Arrival Picking Method for Ultrasound Sound-Speed Tomography

Ultrasound sound-speed tomography (USST) has shown great prospects for breast cancer diagnosis due to its advantages of nonradiation, low cost, 3-D breast images, and quantitative indicators. However, the reconstruction quality of USST is highly dependent on the first-arrival picking of the transmission wave. Traditional first-arrival picking methods have low accuracy and noise robustness. To improve the accuracy and robustness, we introduced a self-attention mechanism into the bidirectional long short-term memory (BLSTM) network and proposed the self-attention BLSTM (SAT-BLSTM) network. The proposed method predicts the probability of the first-arrival time and selects the time with maximum probability. A numerical simulation and prototype experiment were conducted. In the numerical simulation, the proposed SAT-BLSTM showed the best results. For signal-to-noise ratios (SNRs) of 50, 30, and 15 dB, the mean absolute errors (MAEs) were 48, 49, and 76 ns, respectively. The BLSTM had the second-best results, with MAEs of 55, 56, and 85 ns, respectively. The MAEs of the Akaike information criterion (AIC) method were 57, 296, and 489 ns, respectively. In the prototype experiment, the MAEs of the SAT-BLSTM, the BLSTM, and the AIC were 94, 111, and 410 ns, respectively.

Deep Learning-Based Differentiation between Mucinous Cystic Neoplasm and Serous Cystic Neoplasm in the Pancreas Using Endoscopic Ultrasonography

Mucinous cystic neoplasms (MCN) and serous cystic neoplasms (SCN) account for a large portion of solitary pancreatic cystic neoplasms (PCN). In this study we implemented a convolutional neural network (CNN) model using ResNet50 to differentiate between MCN and SCN. The training data were collected retrospectively from 59 MCN and 49 SCN patients from two different hospitals. Data augmentation was used to enhance the size and quality of training datasets. Fine-tuning training approaches were utilized by adopting the pre-trained model from transfer learning while training selected layers. Testing of the network was conducted by varying the endoscopic ultrasonography (EUS) image sizes and positions to evaluate the network performance for differentiation. The proposed network model achieved up to 82.75% accuracy and a 0.88 (95% CI: 0.817–0.930) area under curve (AUC) score. The performance of the implemented deep learning networks in decision-making using only EUS images is comparable to that of traditional manual decision-making using EUS images along with supporting clinical information. Gradient-weighted class activation mapping (Grad-CAM) confirmed that the network model learned the features from the cyst region accurately. This study proves the feasibility of diagnosing MCN and SCN using a deep learning network model. Further improvement using more datasets is needed.

Deep learning for chest X-ray analysis: A survey

Recent advances in deep learning have led to a promising performance in many medical image analysis tasks. As the most commonly performed radiological exam, chest radiographs are a particularly important modality for which a variety of applications have been researched. The release of multiple, large, publicly available chest X-ray datasets in recent years has encouraged research interest and boosted the number of publications. In this paper, we review all studies using deep learning on chest radiographs published before March 2021, categorizing works by task: image-level prediction (classification and regression), segmentation, localization, image generation and domain adaptation. Detailed descriptions of all publicly available datasets are included and commercial systems in the field are described. A comprehensive discussion of the current state of the art is provided, including caveats on the use of public datasets, the requirements of clinically useful systems and gaps in the current literature.

Multiple U-Net-Based Automatic Segmentations and Radiomics Feature Stability on Ultrasound Images for Patients With Ovarian Cancer

Few studies have reported the reproducibility and stability of ultrasound (US) images based radiomics features obtained from automatic segmentation in oncology. The purpose of this study is to study the accuracy of automatic segmentation algorithms based on multiple U-net models and their effects on radiomics features from US images for patients with ovarian cancer. A total of 469 US images from 127 patients were collected and randomly divided into three groups: training sets (353 images), validation sets (23 images), and test sets (93 images) for automatic segmentation models building. Manual segmentation of target volumes was delineated as ground truth. Automatic segmentations were conducted with U-net, U-net++, U-net with Resnet as the backbone (U-net with Resnet), and CE-Net. A python 3.7.0 and package Pyradiomics 2.2.0 were used to extract radiomic features from the segmented target volumes. The accuracy of automatic segmentations was evaluated by Jaccard similarity coefficient (JSC), dice similarity coefficient (DSC), and average surface distance (ASD). The reliability of radiomics features were evaluated by Pearson correlation and intraclass correlation coefficients (ICC). CE-Net and U-net with Resnet outperformed U-net and U-net++ in accuracy performance by achieving a DSC, JSC, and ASD of 0.87, 0.79, 8.54, and 0.86, 0.78, 10.00, respectively. A total of 97 features were extracted from the delineated target volumes. The average Pearson correlation was 0.86 (95% CI, 0.83–0.89), 0.87 (95% CI, 0.84–0.90), 0.88 (95% CI, 0.86–0.91), and 0.90 (95% CI, 0.88–0.92) for U-net++, U-net, U-net with Resnet, and CE-Net, respectively. The average ICC was 0.84 (95% CI, 0.81–0.87), 0.85 (95% CI, 0.82–0.88), 0.88 (95% CI, 0.85–0.90), and 0.89 (95% CI, 0.86–0.91) for U-net++, U-net, U-net with Resnet, and CE-Net, respectively. CE-Net based segmentation achieved the best radiomics reliability. In conclusion, U-net based automatic segmentation was accurate enough to delineate the target volumes on US images for patients with ovarian cancer. Radiomics features extracted from automatic segmented targets showed good reproducibility and for reliability further radiomics investigations.