HCTNet: A hybrid CNN-transformer network for breast ultrasound image segmentation

AI in Breast Cancer Imaging: An Update and Future Trends

Breast cancer is one of the most common types of cancer affecting women worldwide. Artificial intelligence (AI) is transforming breast cancer imaging by enhancing diagnostic capabilities across multiple imaging modalities including mammography, digital breast tomosynthesis, ultrasound, magnetic resonance imaging, and nuclear medicines techniques. AI is being applied to diverse tasks such as breast lesion detection and classification, risk stratification, molecular subtyping, gene mutation status prediction, and treatment response assessment, with emerging research demonstrating performance levels comparable to or potentially exceeding those of radiologists. The large foundation models are showing remarkable potential in different breast cancer imaging tasks. Self-supervised learning gives an insight into data inherent correlation, and federated learning is an alternative way to maintain data privacy. While promising results have been obtained so far, data standardization from source, large-scale annotated multimodal datasets, and extensive prospective clinical trials are still needed to fully explore and validate deep learning's clinical utility and address the legal and ethical considerations, which will ultimately determine its widespread adoption in breast cancer care. We hereby provide a review of the most up-to-date knowledge on AI in breast cancer imaging.

EfficientNet-Based Attention Residual U-Net With Guided Loss for Breast Tumor Segmentation in Ultrasound Images

OBJECTIVE

Breast cancer poses a major health concern for women globally. Effective segmentation of breast tumors for ultrasound images is crucial for early diagnosis and treatment. Conventional convolutional neural networks have shown promising results in this domain but face challenges due to image complexities and are computationally expensive, limiting their practical application in real-time clinical settings.

METHODS

We propose Eff-AResUNet-GL, a segmentation model using EfficienetNet-B3 as the encoder. This design integrates attention gates in skip connections to focus on significant features and residual blocks in the decoder to retain details and reduce gradient loss. Additionally, guided loss functions are applied at each decoder layer to generate better features, subsequently improving segmentation accuracy.

RESULTS

Experimental results on BUSIS and Dataset B demonstrate that Eff-AResUNet-GL achieves superior performance and computational efficiency compared to state-of-the-art models, showing robustness in handling complex breast ultrasound images.

CONCLUSION

Eff-AResUNet-GL offers a practical, high-performing solution for breast tumor segmentation, demonstrating potential clinical through improved segmentation accuracy and efficiency.

Transformer and Attention-Based Architectures for Segmentation of Coronary Arterial Walls in Intravascular Ultrasound: A Narrative Review

Background: The leading global cause of death is coronary artery disease (CAD), necessitating early and precise diagnosis. Intravascular ultrasound (IVUS) is a sophisticated imaging technique that provides detailed visualization of coronary arteries. However, the methods for segmenting walls in the IVUS scan into internal wall structures and quantifying plaque are still evolving. This study explores the use of transformers and attention-based models to improve diagnostic accuracy for wall segmentation in IVUS scans. Thus, the objective is to explore the application of transformer models for wall segmentation in IVUS scans to assess their inherent biases in artificial intelligence systems for improving diagnostic accuracy. Methods: By employing the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework, we pinpointed and examined the top strategies for coronary wall segmentation using transformer-based techniques, assessing their traits, scientific soundness, and clinical relevancy. Coronary artery wall thickness is determined by using the boundaries (inner: lumen-intima and outer: media-adventitia) through cross-sectional IVUS scans. Additionally, it is the first to investigate biases in deep learning (DL) systems that are associated with IVUS scan wall segmentation. Finally, the study incorporates explainable AI (XAI) concepts into the DL structure for IVUS scan wall segmentation. Findings: Because of its capacity to automatically extract features at numerous scales in encoders, rebuild segmented pictures via decoders, and fuse variations through skip connections, the UNet and transformer-based model stands out as an efficient technique for segmenting coronary walls in IVUS scans. Conclusions: The investigation underscores a deficiency in incentives for embracing XAI and pruned AI (PAI) models, with no UNet systems attaining a bias-free configuration. Shifting from theoretical study to practical usage is crucial to bolstering clinical evaluation and deployment.

Progress in the Application of Artificial Intelligence in Ultrasound-Assisted Medical Diagnosis

The integration of artificial intelligence (AI) into ultrasound medicine has revolutionized medical imaging, enhancing diagnostic accuracy and clinical workflows. This review focuses on the applications, challenges, and future directions of AI technologies, particularly machine learning (ML) and its subset, deep learning (DL), in ultrasound diagnostics. By leveraging advanced algorithms such as convolutional neural networks (CNNs), AI has significantly improved image acquisition, quality assessment, and objective disease diagnosis. AI-driven solutions now facilitate automated image analysis, intelligent diagnostic assistance, and medical education, enabling precise lesion detection across various organs while reducing physician workload. AI's error detection capabilities further enhance diagnostic accuracy. Looking ahead, the integration of AI with ultrasound is expected to deepen, promoting trends in standardization, personalized treatment, and intelligent healthcare, particularly in underserved areas. Despite its potential, comprehensive assessments of AI's diagnostic accuracy and ethical implications remain limited, necessitating rigorous evaluations to ensure effectiveness in clinical practice. This review provides a systematic evaluation of AI technologies in ultrasound medicine, highlighting their transformative potential to improve global healthcare outcomes.

BMSMM-Net: A Bone Metastasis Segmentation Framework Based on Mamba and Multiperspective Extraction

RATIONALE AND OBJECTIVES

Metastatic bone tumors significantly reduce patients' quality of life and expedite cancer spread. Traditional diagnostic methods rely on time-consuming manual annotations by radiologists, which are prone to subjectivity. Employing deep learning for rapid, precise segmentation of bone metastases can greatly improve patient outcomes and survival. However, accurate segmentation remains challenging due to the diverse and complex nature of osteoblastic, osteolytic, or mixed lesions.

MATERIALS AND METHODS

In this study, we presented a novel segmentation framework, termed BMSMM-Net, tailored specifically for the detection of bone metastases. The framework integrates our newly proposed Bottleneck Gating Mamba layer (BGM) into the network backbone, enhancing the long-range dependencies in the depth feature maps. Additionally, we designed a Skip-Mamba (SKM) module on the skip connections to facilitate long-range modeling during multi-scale feature fusion. Furthermore, a Multi-Perspective Extraction (MPE) module was employed in the feature extraction phase, utilizing three different sizes of convolutional kernels to enhance sensitivity to bone metastases.

RESULTS

Our framework was evaluated on the BM-Seg dataset through comparative and ablation studies. It achieved F1 scores of 91.07% and 95.17% for segmenting bone metastases and bone regions, respectively, along with mIoU scores of 83.60% and 90.78%, BMSMM-Net provides high-performance segmentation of bone metastases. Additionally, it maintains good computational efficiency compared to existing models.

CONCLUSION

The BMSMM-Net framework, integrating BGM, SKM, and MPE modules, effectively addresses the segmentation challenges of bone metastases. It significantly enhances accuracy, outperforms advanced existing methods, and maintains lower complexity, making it suitable for clinical application.

NMTNet: A Multi-task Deep Learning Network for Joint Segmentation and Classification of Breast Tumors

Segmentation and classification of breast tumors are two critical tasks since they provide significant information for computer-aided breast cancer diagnosis. Combining these tasks leverages their intrinsic relevance to enhance performance, but the variability and complexity of tumor characteristics remain challenging. We propose a novel multi-task deep learning network (NMTNet) for the joint segmentation and classification of breast tumors, which is based on a convolutional neural network (CNN) and U-shaped architecture. It mainly comprises a shared encoder, a multi-scale fusion channel refinement (MFCR) module, a segmentation branch, and a classification branch. First, ResNet18 is used as the backbone network in the encoding part to enhance the feature representation capability. Then, the MFCR module is introduced to enrich the feature depth and diversity. Besides, the segmentation branch combines a lesion region enhancement (LRE) module between the encoder and decoder parts, aiming to capture more detailed texture and edge information of irregular tumors to improve segmentation accuracy. The classification branch incorporates a fine-grained classifier that reuses valuable segmentation information to discriminate between benign and malignant tumors. The proposed NMTNet is evaluated on both ultrasound and magnetic resonance imaging datasets. It achieves segmentation dice scores of 90.30% and 91.50%, and Jaccard indices of 84.70% and 88.10% for each dataset, respectively. And the classification accuracy scores are 87.50% and 99.64% for the corresponding datasets, respectively. Experimental results demonstrate the superiority of NMTNet over state-of-the-art methods on breast tumor segmentation and classification tasks.

Special Issue: Artificial Intelligence in Advanced Medical Imaging

Medical imaging is of great significance in modern medicine and is a crucial part of medical diagnosis [...].

Boundary-aware convolutional attention network for liver segmentation in ultrasound images

Liver ultrasound is widely used in clinical practice due to its advantages of non-invasiveness, non-radiation, and real-time imaging. Accurate segmentation of the liver region in ultrasound images is essential for accelerating the auxiliary diagnosis of liver-related diseases. This paper proposes BACANet, a deep learning algorithm designed for real-time liver ultrasound segmentation. Our approach utilizes a lightweight network backbone for liver feature extraction and incorporates a convolutional attention mechanism to enhance the network's ability to capture global contextual information. To improve early localization of liver boundaries, we developed a selective large kernel convolution module for boundary feature extraction and introduced explicit liver boundary supervision. Additionally, we designed an enhanced attention gate to efficiently convey liver body and boundary features to the decoder to enhance the feature representation capability. Experimental results across multiple datasets demonstrate that BACANet effectively completes the task of liver ultrasound segmentation, achieving a balance between inference speed and segmentation accuracy. On a public dataset, BACANet achieved a DSC of 0.921 and an IOU of 0.854. On a private test dataset, BACANet achieved a DSC of 0.950 and an IOU of 0.907, with an inference time of approximately 0.32 s per image on a CPU processor.

Spectral analysis enhanced net (SAE-Net) to classify breast lesions with BI-RADS category 4 or higher

Early ultrasound screening for breast cancer reduces mortality significantly. The main evaluation criterion for breast ultrasound screening is the Breast Imaging-Reporting and Data System (BI-RADS), which categorizes breast lesions into categories 0-6 based on ultrasound grayscale images. Due to the limitations of ultrasound grayscale imaging, lesions with categories 4 and 5 necessitate additional biopsy for the confirmation of benign or malignant status. In this paper, the SAE-Net was proposed to combine the tissue microstructure information with the morphological information, thus improving the identification of high-grade breast lesions. The SAE-Net consists of a grayscale image branch and a spectral pattern branch. The grayscale image branch used the classical deep learning backbone model to learn the image morphological features from grayscale images, while the spectral pattern branch is designed to learn the microstructure features from ultrasound radio frequency (RF) signals. Our experimental results show that the best SAE-Net model has an area under the receiver operating characteristic curve (AUROC) of 12% higher and a Youden index of 19% higher than the single backbone model. These results demonstrate the effectiveness of our method, which potentially optimizes biopsy exemption and diagnostic efficiency.

SymTC: A symbiotic Transformer-CNN net for instance segmentation of lumbar spine MRI

Intervertebral disc disease, a prevalent ailment, frequently leads to intermittent or persistent low back pain, and diagnosing and assessing of this disease rely on accurate measurement of vertebral bone and intervertebral disc geometries from lumbar MR images. Deep neural network (DNN) models may assist clinicians with more efficient image segmentation of individual instances (discs and vertebrae) of the lumbar spine in an automated way, which is termed as instance image segmentation. In this work, we proposed SymTC, an innovative lumbar spine MR image segmentation model that combines the strengths of Transformer and Convolutional Neural Network (CNN). Specifically, we designed a parallel dual-path architecture to merge CNN layers and Transformer layers, and we integrated a novel position embedding into the self-attention module of Transformer, enhancing the utilization of positional information for more accurate segmentation. To further improve model performance, we introduced a new data synthesis technique to create synthetic yet realistic MR image dataset, named SSMSpine, which is made publicly available. We evaluated our SymTC and the other 16 representative image segmentation models on our private in-house dataset and public SSMSpine dataset, using two metrics, Dice Similarity Coefficient and the 95th percentile Hausdorff Distance. The results indicate that SymTC surpasses the other 16 methods, achieving the highest dice score of 96.169 % for segmenting vertebral bones and intervertebral discs on the SSMSpine dataset. The SymTC code and SSMSpine dataset are publicly available at https://github.com/jiasongchen/SymTC.

NeighborNet: Learning Intra- and Inter-Image Pixel Neighbor Representation for Breast Lesion Segmentation

Breast lesion segmentation from ultrasound images is essential in computer-aided breast cancer diagnosis. To alleviate the problems of blurry lesion boundaries and irregular morphologies, common practices combine CNN and attention to integrate global and local information. However, previous methods use two independent modules to extract global and local features separately, such feature-wise inflexible integration ignores the semantic gap between them, resulting in representation redundancy/insufficiency and undesirable restrictions in clinic practices. Moreover, medical images are highly similar to each other due to the imaging methods and human tissues, but the captured global information by transformer-based methods in the medical domain is limited within images, the semantic relations and common knowledge across images are largely ignored. To alleviate the above problems, in the neighbor view, this paper develops a pixel neighbor representation learning method (NeighborNet) to flexibly integrate global and local context within and across images for lesion morphology and boundary modeling. Concretely, we design two neighbor layers to investigate two properties (i.e., number and distribution) of neighbors. The neighbor number for each pixel is not fixed but determined by itself. The neighbor distribution is extended from one image to all images in the datasets. With the two properties, for each pixel at each feature level, the proposed NeighborNet can evolve into the transformer or degenerate into the CNN for adaptive context representation learning to cope with the irregular lesion morphologies and blurry boundaries. The state-of-the-art performances on three ultrasound datasets prove the effectiveness of the proposed NeighborNet.

Automated contouring of CTV and OARs in planning CT scans using novel hybrid convolution-transformer networks for prostate cancer radiotherapy

PURPOSE OBJECTIVE(S)

Manual contouring of the prostate region in planning computed tomography (CT) images is a challenging task due to factors such as low contrast in soft tissues, inter- and intra-observer variability, and variations in organ size and shape. Consequently, the use of automated contouring methods can offer significant advantages. In this study, we aimed to investigate automated male pelvic multi-organ contouring in multi-center planning CT images using a hybrid convolutional neural network-vision transformer (CNN-ViT) that combines convolutional and ViT techniques.

MATERIALS/METHODS

We used retrospective data from 104 localized prostate cancer patients, with delineations of the clinical target volume (CTV) and critical organs at risk (OAR) for external beam radiotherapy. We introduced a novel attention-based fusion module that merges detailed features extracted through convolution with the global features obtained through the ViT.

RESULTS

The average dice similarity coefficients (DSCs) achieved by VGG16-UNet-ViT for the prostate, bladder, rectum, right femoral head (RFH), and left femoral head (LFH) were 91.75%, 95.32%, 87.00%, 96.30%, and 96.34%, respectively. Experiments conducted on multi-center planning CT images indicate that combining the ViT structure with the CNN network resulted in superior performance for all organs compared to pure CNN and transformer architectures. Furthermore, the proposed method achieves more precise contours compared to state-of-the-art techniques.

CONCLUSION

Results demonstrate that integrating ViT into CNN architectures significantly improves segmentation performance. These results show promise as a reliable and efficient tool to facilitate prostate radiotherapy treatment planning.

A deep learning-based method for the detection and segmentation of breast masses in ultrasound images

Objective.Automated detection and segmentation of breast masses in ultrasound images are critical for breast cancer diagnosis, but remain challenging due to limited image quality and complex breast tissues. This study aims to develop a deep learning-based method that enables accurate breast mass detection and segmentation in ultrasound images.Approach.A novel convolutional neural network-based framework that combines the You Only Look Once (YOLO) v5 network and the Global-Local (GOLO) strategy was developed. First, YOLOv5 was applied to locate the mass regions of interest (ROIs). Second, a Global Local-Connected Multi-Scale Selection (GOLO-CMSS) network was developed to segment the masses. The GOLO-CMSS operated on both the entire images globally and mass ROIs locally, and then integrated the two branches for a final segmentation output. Particularly, in global branch, CMSS applied Multi-Scale Selection (MSS) modules to automatically adjust the receptive fields, and Multi-Input (MLI) modules to enable fusion of shallow and deep features at different resolutions. The USTC dataset containing 28 477 breast ultrasound images was collected for training and test. The proposed method was also tested on three public datasets, UDIAT, BUSI and TUH. The segmentation performance of GOLO-CMSS was compared with other networks and three experienced radiologists.Main results.YOLOv5 outperformed other detection models with average precisions of 99.41%, 95.15%, 93.69% and 96.42% on the USTC, UDIAT, BUSI and TUH datasets, respectively. The proposed GOLO-CMSS showed superior segmentation performance over other state-of-the-art networks, with Dice similarity coefficients (DSCs) of 93.19%, 88.56%, 87.58% and 90.37% on the USTC, UDIAT, BUSI and TUH datasets, respectively. The mean DSC between GOLO-CMSS and each radiologist was significantly better than that between radiologists (p< 0.001).Significance.Our proposed method can accurately detect and segment breast masses with a decent performance comparable to radiologists, highlighting its great potential for clinical implementation in breast ultrasound examination.

tFUSFormer: Physics-Guided Super-Resolution Transformer for Simulation of Transcranial Focused Ultrasound Propagation in Brain Stimulation

Transcranial focused ultrasound (tFUS) has emerged as a new mode of non-invasive brain stimulation (NIBS), with its exquisite spatial precision and capacity to reach the deep regions of the brain. The placement of the acoustic focus onto the desired part of the brain is critical for successful tFUS procedures; however, acoustic wave propagation is severely affected by the skull, distorting the focal location/shape and the pressure level. High-resolution (HR) numerical simulation allows for monitoring of acoustic pressure within the skull but with a considerable computational burden. To address this challenge, we employed a 4× super-resolution (SR) Swin Transformer method to improve the precision of estimating tFUS acoustic pressure field, targeting operator-defined brain areas. The training datasets were obtained through numerical simulations at both ultra-low (2.0 [Formula: see text]) and high (0.5 [Formula: see text]) resolutions, conducted on in vivo CT images of 12 human skulls. Our multivariable datasets, which incorporate physical properties of the acoustic pressure field, wave velocity, and skull CT images, were utilized to train three-dimensional SR models. We found that our method yielded 87.99 ± 4.28% accuracy in terms of focal volume conformity under foreseen skull data, and accuracy of 82.32 ± 5.83% for unforeseen skulls, respectively. Moreover, a significant improvement of 99.4% in computational efficiency compared to the traditional 0.5 [Formula: see text] HR numerical simulation was shown. The presented technique, when adopted in guiding the placement of the FUS transducer to engage specific brain targets, holds great potential in enhancing the safety and effectiveness of tFUS therapy.

Weakly Supervised Breast Ultrasound Image Segmentation Based on Image Selection

Automatic segmentation in Breast Ultrasound (BUS) imaging is vital to BUS computer-aided diagnostic systems. Fully supervised learning approaches can attain high accuracy, yet they depend on pixel-level annotations that are challenging to obtain. As an alternative, weakly supervised learning methods offer a way to lessen the dependency on extensive annotation requirements. Existing weakly supervised learning methods are typically trained on the entire dataset, but not all samples are effective in training a robust image segmentation model. To overcome this challenge, we have developed a new weakly supervised learning approach for BUS image segmentation. Our framework includes three key contributions: 1) A novel image selection method using Class Activation Maps is proposed to identify high-quality candidates for generating pseudo-segmentation labels; 2) The 'Segment Anything' is utilized for pseudo-label generation; 3) A segmentation model is trained using a Mean Teacher method, incorporating both pseudo-labeled and non-labeled images. The proposed framework is evaluated on a public BUS image dataset and achieves an Intersection over Union score that is 82.9% of what is attained by fully supervised methods.

Improving the Automated Diagnosis of Breast Cancer with Mesh Reconstruction of Ultrasound Images Incorporating 3D Mesh Features and a Graph Attention Network

This study proposes a novel approach for breast tumor classification from ultrasound images into benign and malignant by converting the region of interest (ROI) of a 2D ultrasound image into a 3D representation using the point-e system, allowing for in-depth analysis of underlying characteristics. Instead of relying solely on 2D imaging features, this method extracts 3D mesh features that describe tumor patterns more precisely. Ten informative and medically relevant mesh features are extracted and assessed with two feature selection techniques. Additionally, a feature pattern analysis has been conducted to determine the feature's significance. A feature table with dimensions of 445 × 12 is generated and a graph is constructed, considering the rows as nodes and the relationships among the nodes as edges. The Spearman correlation coefficient method is employed to identify edges between the strongly connected nodes (with a correlation score greater than or equal to 0.7), resulting in a graph containing 56,054 edges and 445 nodes. A graph attention network (GAT) is proposed for the classification task and the model is optimized with an ablation study, resulting in the highest accuracy of 99.34%. The performance of the proposed model is compared with ten machine learning (ML) models and one-dimensional convolutional neural network where the test accuracy of these models ranges from 73 to 91%. Our novel 3D mesh-based approach, coupled with the GAT, yields promising performance for breast tumor classification, outperforming traditional models, and has the potential to reduce time and effort of radiologists providing a reliable diagnostic system.

MEF-UNet: An end-to-end ultrasound image segmentation algorithm based on multi-scale feature extraction and fusion

Ultrasound image segmentation is a challenging task due to the complexity of lesion types, fuzzy boundaries, and low-contrast images along with the presence of noises and artifacts. To address these issues, we propose an end-to-end multi-scale feature extraction and fusion network (MEF-UNet) for the automatic segmentation of ultrasound images. Specifically, we first design a selective feature extraction encoder, including detail extraction stage and structure extraction stage, to precisely capture the edge details and overall shape features of the lesions. In order to enhance the representation capacity of contextual information, we develop a context information storage module in the skip-connection section, responsible for integrating information from adjacent two-layer feature maps. In addition, we design a multi-scale feature fusion module in the decoder section to merge feature maps with different scales. Experimental results indicate that our MEF-UNet can significantly improve the segmentation results in both quantitative analysis and visual effects.

DAU-Net: Dual attention-aided U-Net for segmenting tumor in breast ultrasound images

Breast cancer remains a critical global concern, underscoring the urgent need for early detection and accurate diagnosis to improve survival rates among women. Recent developments in deep learning have shown promising potential for computer-aided detection (CAD) systems to address this challenge. In this study, a novel segmentation method based on deep learning is designed to detect tumors in breast ultrasound images. Our proposed approach combines two powerful attention mechanisms: the novel Positional Convolutional Block Attention Module (PCBAM) and Shifted Window Attention (SWA), integrated into a Residual U-Net model. The PCBAM enhances the Convolutional Block Attention Module (CBAM) by incorporating the Positional Attention Module (PAM), thereby improving the contextual information captured by CBAM and enhancing the model's ability to capture spatial relationships within local features. Additionally, we employ SWA within the bottleneck layer of the Residual U-Net to further enhance the model's performance. To evaluate our approach, we perform experiments using two widely used datasets of breast ultrasound images and the obtained results demonstrate its capability in accurately detecting tumors. Our approach achieves state-of-the-art performance with dice score of 74.23% and 78.58% on BUSI and UDIAT datasets, respectively in segmenting the breast tumor region, showcasing its potential to help with precise tumor detection. By leveraging the power of deep learning and integrating innovative attention mechanisms, our study contributes to the ongoing efforts to improve breast cancer detection and ultimately enhance women's survival rates. The source code of our work can be found here: https://github.com/AyushRoy2001/DAUNet.

Multi-task learning for segmentation and classification of breast tumors from ultrasound images

Segmentation and classification of breast tumors are critical components of breast ultrasound (BUS) computer-aided diagnosis (CAD), which significantly improves the diagnostic accuracy of breast cancer. However, the characteristics of tumor regions in BUS images, such as non-uniform intensity distributions, ambiguous or missing boundaries, and varying tumor shapes and sizes, pose significant challenges to automated segmentation and classification solutions. Many previous studies have proposed multi-task learning methods to jointly tackle tumor segmentation and classification by sharing the features extracted by the encoder. Unfortunately, this often introduces redundant or misleading information, which hinders effective feature exploitation and adversely affects performance. To address this issue, we present ACSNet, a novel multi-task learning network designed to optimize tumor segmentation and classification in BUS images. The segmentation network incorporates a novel gate unit to allow optimal transfer of valuable contextual information from the encoder to the decoder. In addition, we develop the Deformable Spatial Attention Module (DSAModule) to improve segmentation accuracy by overcoming the limitations of conventional convolution in dealing with morphological variations of tumors. In the classification branch, multi-scale feature extraction and channel attention mechanisms are integrated to discriminate between benign and malignant breast tumors. Experiments on two publicly available BUS datasets demonstrate that ACSNet not only outperforms mainstream multi-task learning methods for both breast tumor segmentation and classification tasks, but also achieves state-of-the-art results for BUS tumor segmentation. Code and models are available at https://github.com/qqhe-frank/BUS-segmentation-and-classification.git.

Automated analysis of pectoralis major thickness in pec-fly exercises: evolving from manual measurement to deep learning techniques

This study addresses a limitation of prior research on pectoralis major (PMaj) thickness changes during the pectoralis fly exercise using a wearable ultrasound imaging setup. Although previous studies used manual measurement and subjective evaluation, it is important to acknowledge the subsequent limitations of automating widespread applications. We then employed a deep learning model for image segmentation and automated measurement to solve the problem and study the additional quantitative supplementary information that could be provided. Our results revealed increased PMaj thickness changes in the coronal plane within the probe detection region when real-time ultrasound imaging (RUSI) visual biofeedback was incorporated, regardless of load intensity (50% or 80% of one-repetition maximum). Additionally, participants showed uniform thickness changes in the PMaj in response to enhanced RUSI biofeedback. Notably, the differences in PMaj thickness changes between load intensities were reduced by RUSI biofeedback, suggesting altered muscle activation strategies. We identified the optimal measurement location for the maximal PMaj thickness close to the rib end and emphasized the lightweight applicability of our model for fitness training and muscle assessment. Further studies can refine load intensities, investigate diverse parameters, and employ different network models to enhance accuracy. This study contributes to our understanding of the effects of muscle physiology and exercise training.

Deep Learning Detection and Segmentation of Facet Joints in Ultrasound Images Based on Convolutional Neural Networks and Enhanced Data Annotation

The facet joint injection is the most common procedure used to release lower back pain. In this paper, we proposed a deep learning method for detecting and segmenting facet joints in ultrasound images based on convolutional neural networks (CNNs) and enhanced data annotation. In the enhanced data annotation, a facet joint was considered as the first target and the ventral complex as the second target to improve the capability of CNNs in recognizing the facet joint. A total of 300 cases of patients undergoing pain treatment were included. The ultrasound images were captured and labeled by two professional anesthesiologists, and then augmented to train a deep learning model based on the Mask Region-based CNN (Mask R-CNN). The performance of the deep learning model was evaluated using the average precision (AP) on the testing sets. The data augmentation and data annotation methods were found to improve the AP. The AP50 for facet joint detection and segmentation was 90.4% and 85.0%, respectively, demonstrating the satisfying performance of the deep learning model. We presented a deep learning method for facet joint detection and segmentation in ultrasound images based on enhanced data annotation and the Mask R-CNN. The feasibility and potential of deep learning techniques in facet joint ultrasound image analysis have been demonstrated.

CoT: a transformer-based method for inferring tumor clonal copy number substructure from scDNA-seq data

Single-cell DNA sequencing (scDNA-seq) has been an effective means to unscramble intra-tumor heterogeneity, while joint inference of tumor clones and their respective copy number profiles remains a challenging task due to the noisy nature of scDNA-seq data. We introduce a new bioinformatics method called CoT for deciphering clonal copy number substructure. The backbone of CoT is a Copy number Transformer autoencoder that leverages multi-head attention mechanism to explore correlations between different genomic regions, and thus capture global features to create latent embeddings for the cells. CoT makes it convenient to first infer cell subpopulations based on the learned embeddings, and then estimate single-cell copy numbers through joint analysis of read counts data for the cells belonging to the same cluster. This exploitation of clonal substructure information in copy number analysis helps to alleviate the effect of read counts non-uniformity, and yield robust estimations of the tumor copy numbers. Performance evaluation on synthetic and real datasets showcases that CoT outperforms the state of the arts, and is highly useful for deciphering clonal copy number substructure.

Ultrasound Image Analysis with Vision Transformers-Review

Ultrasound (US) has become a widely used imaging modality in clinical practice, characterized by its rapidly evolving technology, advantages, and unique challenges, such as a low imaging quality and high variability. There is a need to develop advanced automatic US image analysis methods to enhance its diagnostic accuracy and objectivity. Vision transformers, a recent innovation in machine learning, have demonstrated significant potential in various research fields, including general image analysis and computer vision, due to their capacity to process large datasets and learn complex patterns. Their suitability for automatic US image analysis tasks, such as classification, detection, and segmentation, has been recognized. This review provides an introduction to vision transformers and discusses their applications in specific US image analysis tasks, while also addressing the open challenges and potential future trends in their application in medical US image analysis. Vision transformers have shown promise in enhancing the accuracy and efficiency of ultrasound image analysis and are expected to play an increasingly important role in the diagnosis and treatment of medical conditions using ultrasound imaging as technology progresses.

LumVertCancNet: A novel 3D lumbar vertebral body cancellous bone location and segmentation method based on hybrid Swin-transformer

Lumbar vertebral body cancellous bone location and segmentation is crucial in an automated lumbar spine processing pipeline. Accurate and reliable analysis of lumbar spine image is expected to advantage practical medical diagnosis and population-based analysis of bone strength. However, the design of automated algorithms for lumbar spine processing is demanding due to significant anatomical variations and scarcity of publicly available data. In recent years, convolutional neural network (CNN) and vision transformers (Vits) have been the de facto standard in medical image segmentation. Although adept at capturing global features, the inherent bias of locality and weight sharing of CNN constrains its capacity to model long-range dependency. In contrast, Vits excel at long-range dependency modeling, but they may not generalize well with limited datasets due to the lack of inductive biases inherent to CNN. In this paper, we propose a deep learning-based two-stage coarse-to-fine solution to address the problem of automatic location and segmentation of lumbar vertebral body cancellous bone. Specifically, in the first stage, a Swin-transformer based model is applied to predict the heatmap of lumbar vertebral body centroids. Considering the characteristic anatomical structure of lumbar spine, we propose a novel loss function called LumAnatomy loss, which enforces the order and bend of the predicted vertebral body centroids. To inherit the excellence of CNN and Vits while preventing their respective limitations, in the second stage, we propose an encoder-decoder network to segment the identified lumbar vertebral body cancellous bone, which consists of two parallel encoders, i.e., a Swin-transformer encoder and a CNN encoder. To enhance the combination of CNNs and Vits, we propose a novel multi-scale attention feature fusion module (MSA-FFM), which address issues that arise when fusing features given at different encoders. To tackle the issue of lack of data, we raise the first large-scale lumbar vertebral body cancellous bone segmentation dataset called LumVBCanSeg containing a total of 185 CT scans annotated at voxel level by 3 physicians. Extensive experimental results on the LumVBCanSeg dataset demonstrate the proposed algorithm outperform other state-of-the-art medical image segmentation methods. The data is publicly available at: https://zenodo.org/record/8181250. The implementation of the proposed method is available at: https://github.com/sia405yd/LumVertCancNet.

Precise and Rapid Whole-Head Segmentation from Magnetic Resonance Images of Older Adults using Deep Learning

Whole-head segmentation from Magnetic Resonance Images (MRI) establishes the foundation for individualized computational models using finite element method (FEM). This foundation paves the path for computer-aided solutions in fields, particularly in non-invasive brain stimulation. Most current automatic head segmentation tools are developed using healthy young adults. Thus, they may neglect the older population that is more prone to age-related structural decline such as brain atrophy. In this work, we present a new deep learning method called GRACE, which stands for General, Rapid, And Comprehensive whole-hEad tissue segmentation. GRACE is trained and validated on a novel dataset that consists of 177 manually corrected MR-derived reference segmentations that have undergone meticulous manual review. Each T1-weighted MRI volume is segmented into 11 tissue types, including white matter, grey matter, eyes, cerebrospinal fluid, air, blood vessel, cancellous bone, cortical bone, skin, fat, and muscle. To the best of our knowledge, this work contains the largest manually corrected dataset to date in terms of number of MRIs and segmented tissues. GRACE outperforms five freely available software tools and a traditional 3D U-Net on a five-tissue segmentation task. On this task, GRACE achieves an average Hausdorff Distance of 0.21, which exceeds the runner-up at an average Hausdorff Distance of 0.36. GRACE can segment a whole-head MRI in about 3 seconds, while the fastest software tool takes about 3 minutes. In summary, GRACE segments a spectrum of tissue types from older adults T1-MRI scans at favorable accuracy and speed. The trained GRACE model is optimized on older adult heads to enable high-precision modeling in age-related brain disorders. To support open science, the GRACE code and trained weights are made available online and open to the research community at https://github.com/lab-smile/GRACE.

A versatile Wavelet-Enhanced CNN-Transformer for improved fluorescence microscopy image restoration

Fluorescence microscopes are indispensable tools for the life science research community. Nevertheless, the presence of optical component limitations, coupled with the maximum photon budget that the specimen can tolerate, inevitably leads to a decline in imaging quality and a lack of useful signals. Therefore, image restoration becomes essential for ensuring high-quality and accurate analyses. This paper presents the Wavelet-Enhanced Convolutional-Transformer (WECT), a novel deep learning technique developed specifically for the purpose of reducing noise in microscopy images and attaining super-resolution. Unlike traditional approaches, WECT integrates wavelet transform and inverse-transform for multi-resolution image decomposition and reconstruction, resulting in an expanded receptive field for the network without compromising information integrity. Subsequently, multiple consecutive parallel CNN-Transformer modules are utilized to collaboratively model local and global dependencies, thus facilitating the extraction of more comprehensive and diversified deep features. In addition, the incorporation of generative adversarial networks (GANs) into WECT enhances its capacity to generate high perceptual quality microscopic images. Extensive experiments have demonstrated that the WECT framework outperforms current state-of-the-art restoration methods on real fluorescence microscopy data under various imaging modalities and conditions, in terms of quantitative and qualitative analysis.

Multi-task approach based on combined CNN-transformer for efficient segmentation and classification of breast tumors in ultrasound images

Accurate segmentation of breast ultrasound (BUS) images is crucial for early diagnosis and treatment of breast cancer. Further, the task of segmenting lesions in BUS images continues to pose significant challenges due to the limitations of convolutional neural networks (CNNs) in capturing long-range dependencies and obtaining global context information. Existing methods relying solely on CNNs have struggled to address these issues. Recently, ConvNeXts have emerged as a promising architecture for CNNs, while transformers have demonstrated outstanding performance in diverse computer vision tasks, including the analysis of medical images. In this paper, we propose a novel breast lesion segmentation network CS-Net that combines the strengths of ConvNeXt and Swin Transformer models to enhance the performance of the U-Net architecture. Our network operates on BUS images and adopts an end-to-end approach to perform segmentation. To address the limitations of CNNs, we design a hybrid encoder that incorporates modified ConvNeXt convolutions and Swin Transformer. Furthermore, to enhance capturing the spatial and channel attention in feature maps we incorporate the Coordinate Attention Module. Second, we design an Encoder-Decoder Features Fusion Module that facilitates the fusion of low-level features from the encoder with high-level semantic features from the decoder during the image reconstruction. Experimental results demonstrate the superiority of our network over state-of-the-art image segmentation methods for BUS lesions segmentation.

Monitoring response to neoadjuvant therapy for breast cancer in all treatment phases using an ultrasound deep learning model

Purpose

The aim of this study was to investigate the value of a deep learning model (DLM) based on breast tumor ultrasound image segmentation in predicting pathological response to neoadjuvant chemotherapy (NAC) in breast cancer.

Methods

The dataset contains a total of 1393 ultrasound images of 913 patients from Renmin Hospital of Wuhan University, of which 956 ultrasound images of 856 patients were used as the training set, and 437 ultrasound images of 57 patients underwent NAC were used as the test set. A U-Net-based end-to-end DLM was developed for automatically tumor segmentation and area calculation. The predictive abilities of the DLM, manual segmentation model (MSM), and two traditional ultrasound measurement methods (longest axis model [LAM] and dual-axis model [DAM]) for pathological complete response (pCR) were compared using changes in tumor size ratios to develop receiver operating characteristic curves.

Results

The average intersection over union value of the DLM was 0.856. The early-stage ultrasound-predicted area under curve (AUC) values of pCR were not significantly different from those of the intermediate and late stages (p< 0.05). The AUCs for MSM, DLM, LAM and DAM were 0.840, 0.756, 0.778 and 0.796, respectively. There was no significant difference in AUC values of the predictive ability of the four models.

Conclusion

Ultrasonography was predictive of pCR in the early stages of NAC. DLM have a similar predictive value to conventional ultrasound for pCR, with an add benefit in effectively improving workflow.

A survey on cancer detection via convolutional neural networks: Current challenges and future directions

Cancer is a condition in which abnormal cells uncontrollably split and damage the body tissues. Hence, detecting cancer at an early stage is highly essential. Currently, medical images play an indispensable role in detecting various cancers; however, manual interpretation of these images by radiologists is observer-dependent, time-consuming, and tedious. An automatic decision-making process is thus an essential need for cancer detection and diagnosis. This paper presents a comprehensive survey on automated cancer detection in various human body organs, namely, the breast, lung, liver, prostate, brain, skin, and colon, using convolutional neural networks (CNN) and medical imaging techniques. It also includes a brief discussion about deep learning based on state-of-the-art cancer detection methods, their outcomes, and the possible medical imaging data used. Eventually, the description of the dataset used for cancer detection, the limitations of the existing solutions, future trends, and challenges in this domain are discussed. The utmost goal of this paper is to provide a piece of comprehensive and insightful information to researchers who have a keen interest in developing CNN-based models for cancer detection.

Transformer-Based Automated Segmentation of the Median Nerve in Ultrasound Videos of Wrist-to-Elbow Region

Segmenting the median nerve is essential for identifying nerve entrapment syndromes, guiding surgical planning and interventions, and furthering understanding of nerve anatomy. This study aims to develop an automated tool that can assist clinicians in localizing and segmenting the median nerve from the wrist, mid-forearm, and elbow in ultrasound videos. This is the first fully automated single deep learning model for accurate segmentation of the median nerve from the wrist to the elbow in ultrasound videos, along with the computation of the cross-sectional area (CSA) of the nerve. The visual transformer architecture, which was originally proposed to detect and classify 41 classes in YouTube videos, was modified to predict the median nerve in every frame of ultrasound videos. This is achieved by modifying the bounding box sequence matching block of the visual transformer. The median nerve segmentation is a binary class prediction, and the entire bipartite matching sequence is eliminated, enabling a direct comparison of the prediction with expert annotation in a frame-by-frame fashion. Model training, validation, and testing were performed on a dataset comprising ultrasound videos collected from 100 subjects, which were partitioned into 80, ten, and ten subjects, respectively. The proposed model was compared with U-Net, U-Net++, Siam U-Net, Attention U-Net, LSTM U-Net, and Trans U-Net. The proposed transformer-based model effectively leveraged the temporal and spatial information present in ultrasound video frames and efficiently segmented the median nerve with an average dice similarity coefficient (DSC) of approximately 94% at the wrist and 84% in the entire forearm region.

CFATransUnet: Channel-wise cross fusion attention and transformer for 2D medical image segmentation

Medical image segmentation faces current challenges in effectively extracting and fusing long-distance and local semantic information, as well as mitigating or eliminating semantic gaps during the encoding and decoding process. To alleviate the above two problems, we propose a new U-shaped network structure, called CFATransUnet, with Transformer and CNN blocks as the backbone network, equipped with Channel-wise Cross Fusion Attention and Transformer (CCFAT) module, containing Channel-wise Cross Fusion Transformer (CCFT) and Channel-wise Cross Fusion Attention (CCFA). Specifically, we use a Transformer and CNN blocks to construct the encoder and decoder for adequate extraction and fusion of long-range and local semantic features. The CCFT module utilizes the self-attention mechanism to reintegrate semantic information from different stages into cross-level global features to reduce the semantic asymmetry between features at different levels. The CCFA module adaptively acquires the importance of each feature channel based on a global perspective in a network learning manner, enhancing effective information grasping and suppressing non-important features to mitigate semantic gaps. The combination of CCFT and CCFA can guide the effective fusion of different levels of features more powerfully with a global perspective. The consistent architecture of the encoder and decoder also alleviates the semantic gap. Experimental results suggest that the proposed CFATransUnet achieves state-of-the-art performance on four datasets. The code is available at https://github.com/CPU0808066/CFATransUnet.

Graph neural network-based breast cancer diagnosis using ultrasound images with optimized graph construction integrating the medically significant features

PURPOSE

An automated computerized approach can aid radiologists in the early diagnosis of breast cancer. In this study, a novel method is proposed for classifying breast tumors into benign and malignant, based on the ultrasound images through a Graph Neural Network (GNN) model utilizing clinically significant features.

METHOD

Ten informative features are extracted from the region of interest (ROI), based on the radiologists' diagnosis markers. The significance of the features is evaluated using density plot and T test statistical analysis method. A feature table is generated where each row represents individual image, considered as node, and the edges between the nodes are denoted by calculating the Spearman correlation coefficient. A graph dataset is generated and fed into the GNN model. The model is configured through ablation study and Bayesian optimization. The optimized model is then evaluated with different correlation thresholds for getting the highest performance with a shallow graph. The performance consistency is validated with k-fold cross validation. The impact of utilizing ROIs and handcrafted features for breast tumor classification is evaluated by comparing the model's performance with Histogram of Oriented Gradients (HOG) descriptor features from the entire ultrasound image. Lastly, a clustering-based analysis is performed to generate a new filtered graph, considering weak and strong relationships of the nodes, based on the similarities.

RESULTS

The results indicate that with a threshold value of 0.95, the GNN model achieves the highest test accuracy of 99.48%, precision and recall of 100%, and F1 score of 99.28%, reducing the number of edges by 85.5%. The GNN model's performance is 86.91%, considering no threshold value for the graph generated from HOG descriptor features. Different threshold values for the Spearman's correlation score are experimented with and the performance is compared. No significant differences are observed between the previous graph and the filtered graph.

CONCLUSION

The proposed approach might aid the radiologists in effective diagnosing and learning tumor pattern of breast cancer.

TBUnet: A Pure Convolutional U-Net Capable of Multifaceted Feature Extraction for Medical Image Segmentation

Many current medical image segmentation methods utilize convolutional neural networks (CNNs), with some extended U-Net-based networks relying on deep feature representations to achieve satisfactory results. However, due to the limited receptive fields of convolutional architectures, they are unable to explicitly model the varying range dependencies present in medical images. Recently, advancements in large kernel convolution have allowed for the extraction of a wider range of low frequency information, making this task more achievable. In this paper, we propose TBUnet for solving the problem of difficult to accurately segment lesions with heterogeneous structures and fuzzy borders, such as melanoma, colon polyps and breast cancer. The TBUnet is a pure convolutional network with three branches for extracting high frequency information, low frequency information, and boundary information, respectively. It is capable of extracting features in various areas. To fuse the feature maps from the three branches, TBUnet presents the FL (fusion layer) module, which is based on threshold and logical operation. We design the FE (feature enhancement) module on the skip-connection to emphasize the fine-grained features. In addition, our method varies the number of input channels in different branches at each stage of the network, so that the relationship between low and high frequency features can be learned. TBUnet yields 91.08 DSC on ISIC-2018 for melanoma segmentation, and achieves better performance than state-of-the-art medical image segmentation methods. Furthermore, experimental results with 82.48 DSC and 89.04 DSC obtained on the BUSI dataset and the Kvasir-SEG dataset show that TBUnet outperforms the advanced segmentation methods. Experiments demonstrate that TBUnet has excellent segmentation performance and generalisation capability.

Recent advancements in artificial intelligence for breast cancer: Image augmentation, segmentation, diagnosis, and prognosis approaches

Breast cancer is a significant global health burden, with increasing morbidity and mortality worldwide. Early screening and accurate diagnosis are crucial for improving prognosis. Radiographic imaging modalities such as digital mammography (DM), digital breast tomosynthesis (DBT), magnetic resonance imaging (MRI), ultrasound (US), and nuclear medicine techniques, are commonly used for breast cancer assessment. And histopathology (HP) serves as the gold standard for confirming malignancy. Artificial intelligence (AI) technologies show great potential for quantitative representation of medical images to effectively assist in segmentation, diagnosis, and prognosis of breast cancer. In this review, we overview the recent advancements of AI technologies for breast cancer, including 1) improving image quality by data augmentation, 2) fast detection and segmentation of breast lesions and diagnosis of malignancy, 3) biological characterization of the cancer such as staging and subtyping by AI-based classification technologies, 4) prediction of clinical outcomes such as metastasis, treatment response, and survival by integrating multi-omics data. Then, we then summarize large-scale databases available to help train robust, generalizable, and reproducible deep learning models. Furthermore, we conclude the challenges faced by AI in real-world applications, including data curating, model interpretability, and practice regulations. Besides, we expect that clinical implementation of AI will provide important guidance for the patient-tailored management.

Automated breast tumor ultrasound image segmentation with hybrid UNet and classification using fine-tuned CNN model

Introduction

Breast cancer stands as the second most deadly form of cancer among women worldwide. Early diagnosis and treatment can significantly mitigate mortality rates.

Purpose

The study aims to classify breast ultrasound images into benign and malignant tumors. This approach involves segmenting the breast's region of interest (ROI) employing an optimized UNet architecture and classifying the ROIs through an optimized shallow CNN model utilizing an ablation study.

Method

Several image processing techniques are utilized to improve image quality by removing text, artifacts, and speckle noise, and statistical analysis is done to check the enhanced image quality is satisfactory. With the processed dataset, the segmentation of breast tumor ROI is carried out, optimizing the UNet model through an ablation study where the architectural configuration and hyperparameters are altered. After obtaining the tumor ROIs from the fine-tuned UNet model (RKO-UNet), an optimized CNN model is employed to classify the tumor into benign and malignant classes. To enhance the CNN model's performance, an ablation study is conducted, coupled with the integration of an attention unit. The model's performance is further assessed by classifying breast cancer with mammogram images.

Result

The proposed classification model (RKONet-13) results in an accuracy of 98.41 %. The performance of the proposed model is further compared with five transfer learning models for both pre-segmented and post-segmented datasets. K-fold cross-validation is done to assess the proposed RKONet-13 model's performance stability. Furthermore, the performance of the proposed model is compared with previous literature, where the proposed model outperforms existing methods, demonstrating its effectiveness in breast cancer diagnosis. Lastly, the model demonstrates its robustness for breast cancer classification, delivering an exceptional performance of 96.21 % on a mammogram dataset.

Conclusion

The efficacy of this study relies on image pre-processing, segmentation with hybrid attention UNet, and classification with fine-tuned robust CNN model. This comprehensive approach aims to determine an effective technique for detecting breast cancer within ultrasound images.

Enhanced Deep-Learning-Based Automatic Left-Femur Segmentation Scheme with Attribute Augmentation

This research proposes augmenting cropped computed tomography (CT) slices with data attributes to enhance the performance of a deep-learning-based automatic left-femur segmentation scheme. The data attribute is the lying position for the left-femur model. In the study, the deep-learning-based automatic left-femur segmentation scheme was trained, validated, and tested using eight categories of CT input datasets for the left femur (F-I-F-VIII). The segmentation performance was assessed by Dice similarity coefficient (DSC) and intersection over union (IoU); and the similarity between the predicted 3D reconstruction images and ground-truth images was determined by spectral angle mapper (SAM) and structural similarity index measure (SSIM). The left-femur segmentation model achieved the highest DSC (88.25%) and IoU (80.85%) under category F-IV (using cropped and augmented CT input datasets with large feature coefficients), with an SAM and SSIM of 0.117-0.215 and 0.701-0.732. The novelty of this research lies in the use of attribute augmentation in medical image preprocessing to enhance the performance of the deep-learning-based automatic left-femur segmentation scheme.

Dual-feature Fusion Attention Network for Small Object Segmentation

Accurate segmentation of medical images is an important step during radiotherapy planning and clinical diagnosis. However, manually marking organ or lesion boundaries is tedious, time-consuming, and prone to error due to subjective variability of radiologist. Automatic segmentation remains a challenging task owing to the variation (in shape and size) across subjects. Moreover, existing convolutional neural networks based methods perform poorly in small medical objects segmentation due to class imbalance and boundary ambiguity. In this paper, we propose a dual feature fusion attention network (DFF-Net) to improve the segmentation accuracy of small objects. It mainly includes two core modules: the dual-branch feature fusion module (DFFM) and the reverse attention context module (RACM). We first extract multi-resolution features by multi-scale feature extractor, then construct DFFM to aggregate the global and local contextual information to achieve information complementarity among features, which provides sufficient guidance for accurate small objects segmentation. Moreover, to alleviate the degradation of segmentation accuracy caused by blurred medical image boundaries, we propose RACM to enhance the edge texture of features. Experimental results on datasets NPC, ACDC, and Polyp demonstrate that our proposed method has fewer parameters, faster inference, and lower model complexity, and achieves better accuracy than more state-of-the-art methods.