Attention-Guided Deep Neural Network With Multi-Scale Feature Fusion for Liver Vessel Segmentation

Diagnosis assistant for liver cancer utilizing a large language model with three types of knowledge

Objective.Liver cancer has a high incidence rate, but experienced doctors are lacking in primary healthcare settings. The development of large models offers new possibilities for diagnosis. However, in liver cancer diagnosis, large models face certain limitations, such as insufficient understanding of specific medical images, inadequate consideration of liver vessel factors, and inaccuracies in reasoning logic. Therefore, this study proposes a diagnostic assistance tool specific to liver cancer to enhance the diagnostic capabilities of primary care doctors.Approach.A liver cancer diagnosis framework combining large and small models is proposed. A more accurate model for liver tumor segmentation and a more precise model for liver vessel segmentation are developed. The features extracted from the segmentation results of the small models are combined with the patient's medical records and then provided to the large model. The large model employs chain of thought prompts to simulate expert diagnostic reasoning and uses Retrieval-Augmented Generation to provide reliable answers based on trusted medical knowledge and cases.Main results.In the small model part, the proposed liver tumor and liver vessel segmentation methods achieve improved performance. In the large model part, this approach receives higher evaluation scores from doctors when analyzing patient imaging and medical records.Significance.First, a diagnostic framework combining small models and large models is proposed to optimize the liver cancer diagnosis process. Second, two segmentation models are introduced to compensate for the large model's shortcomings in extracting semantic information from images. Third, by simulating doctors' reasoning and integrating trusted knowledge, the framework enhances the reliability and interpretability of the large model's responses while reducing hallucination phenomena.

VSNet: Vessel Structure-aware Network for hepatic and portal vein segmentation

Identifying and segmenting hepatic and portal veins (two predominant vascular systems in the liver, from CT scans) play a crucial role for clinicians in preoperative planning for treatment strategies. However, existing segmentation models often struggle to capture fine details of minor veins. In this article, we introduce Vessel Structure-aware Network (VSNet), a multi-task learning model with vessel-growing decoder, to address the challenge. VSNet excels at accurate segmentation by capturing the topological features of both minor veins while preserving correct connectivity from minor vessels to trucks. We also build and publish the largest dataset (303 cases) for hepatic and portal vessel segmentation. Through comprehensive experiments, we demonstrate that VSNet achieves the best Dice for hepatic vein of 0.824 and portal vein of 0.807 on our proposed dataset, significantly outperforming other popular segmentation models. The source code and dataset are publicly available at https://github.com/XXYZB/VSNet.

Diagnosis of Lung Cancer Using Endobronchial Ultrasonography Image Based on Multi-Scale Image and Multi-Feature Fusion Framework

Lung cancer is the leading cause of cancer-related deaths globally and ranks among the most common cancer types. Given its low overall five-year survival rate, early diagnosis and timely treatment are essential to improving patient outcomes. In recent years, advances in computer technology have enabled artificial intelligence to make groundbreaking progress in imaging-based lung cancer diagnosis. The primary aim of this study is to develop a computer-aided diagnosis (CAD) system for lung cancer using endobronchial ultrasonography (EBUS) images and deep learning algorithms to facilitate early detection and improve patient survival rates. We propose M3-Net, which is a multi-branch framework that integrates multiple features through an attention-based mechanism, enhancing diagnostic performance by providing more comprehensive information for lung cancer assessment. The framework was validated on a dataset of 95 patient cases, including 13 benign and 82 malignant cases. The dataset comprises 1140 EBUS images, with 540 images used for training, and 300 images each for the validation and test sets. The evaluation yielded the following results: accuracy of 0.76, F1-score of 0.75, AUC of 0.83, PPV of 0.80, NPV of 0.75, sensitivity of 0.72, and specificity of 0.80. These findings indicate that the proposed attention-based multi-feature fusion framework holds significant potential in assisting with lung cancer diagnosis.

TransRAUNet: A Deep Neural Network with Reverse Attention Module Using HU Windowing Augmentation for Robust Liver Vessel Segmentation in Full Resolution of CT Images

Background: Liver cancer has a high mortality rate worldwide, and clinicians segment liver vessels in CT images before surgical procedures. However, liver vessels have a complex structure, and the segmentation process is conducted manually, so it is time-consuming and labor-intensive. Consequently, it would be extremely useful to develop a deep learning-based automatic liver vessel segmentation method. Method: As a segmentation method, UNet is widely used as a baseline, and a multi-scale block or attention module has been introduced to extract context information. In recent machine learning efforts, not only has the global context extraction been improved by introducing Transformer, but a method to reinforce the edge area has been proposed. However, the data preprocessing step still commonly uses general augmentation methods, such as flip, rotation, and mirroring, so it does not perform robustly on images of varying brightness or contrast levels. We propose a method of applying image augmentation with different HU windowing values. In addition, to minimize the false negative area, we propose TransRAUNet, which introduces a reverse attention module (RAM) that can focus edge information to the baseline TransUNet. The proposed architecture solves context loss for small vessels by applying edge module (RAM) in the upsampling phase. It can also generate semantic feature maps that allows it to learn edge, global context, and detail location by combining high-level edge and low-level context features. Results: In the 3Dricadb dataset, the proposed model achieved a DSC of 0.948 and a sensitivity of 0.944 in liver vessel segmentation. This study demonstrated that the proposed augmentation method is effective and robust by comparisons with the model without augmentation and with the general augmentation method. Additionally, an ablation study showed that RAM has improved segmentation performance compared to TransUNet. Compared to prevailing state-of-the-art methods, the proposed model showed the best performance for liver vessel segmentation. Conclusions: TransRAUnet is expected to serve as a navigation aid for liver resection surgery through accurate liver vessel and tumor segmentation.

Detail-sensitive 3D-UNet for pulmonary airway segmentation from CT images

The segmentation of airway from computed tomography (CT) images plays a vital role in pulmonary disease diagnosis, evaluation, surgical planning, and treatment. Nevertheless, it is still challenging for current methods to handle distal thin and low-contrast airways, leading to mis-segmentation issues. This paper proposes a detail-sensitive 3D-UNet (DS-3D-UNet) that incorporates two new modules into 3D-UNet to segment airways accurately from CT images. The feature recalibration module is designed to give more attention to the foreground airway features through a new attention mechanism. The detail extractor module aims to restore multi-scale detailed features by fusion of features at different levels. Extensive experiments were conducted on the ATM'22 challenge dataset composed of 300 CT scans with airway annotations to evaluate its performance. Quantitative comparisons prove that the proposed model achieves the best performance in terms of Dice similarity coefficient (92.6%) and Intersection over Union (86.3%), outperforming other state-of-the-art methods. Qualitative comparisons further exhibit the superior performance of our method in segmenting thin and confused distal bronchi. The proposed model could provide important references for the diagnosis and treatment of pulmonary diseases, holding promising prospects in the field of digital medicine. Codes are available at https://github.com/nighlevil/DS-3D-UNet/tree/master .

A multi-scale, multi-task fusion UNet model for accurate breast tumor segmentation

BACKGROUND AND OBJECTIVE

Breast cancer is the most common cancer type among women worldwide and a leading cause of female death. Accurately interpreting these complex tumors, involving small size and morphology, requires a significant amount of expertise and time. Developing a breast tumor segmentation model to assist clinicians in treatment, therefore, holds great practical significance.

METHODS

We propose a multi-scale, multi-task model framework named MTF-UNet. Firstly, we differ from the common approach of using different convolution kernel sizes to extract multi-scale features, and instead use the same convolution kernel size with different numbers of convolutions to obtain multi-scale, multi-level features. Additionally, to better integrate features from different levels and sizes, we extract a new multi-branch feature fusion block (ADF). This block differs from using channel and spatial attention to fuse features, but considers fusion weights between various branches. Secondly, we propose to use the number of pixels predicted to be related to tumors and background to assist segmentation, which is different from the conventional approach of using classification tasks to assist segmentation.

RESULTS

We conducted extensive experiments on our proprietary DCE-MRI dataset, as well as two public datasets (BUSI and Kvasir-SEG). In the aforementioned datasets, our model achieved excellent MIoU scores of 90.4516%, 89.8408%, and 92.8431% on the respective test sets. Furthermore, our ablation study has demonstrated the efficacy of each component and the effective integration of our auxiliary prediction branch into other models.

CONCLUSION

Through comprehensive experiments and comparisons with other algorithms, the effectiveness, adaptability, and robustness of our proposed method have been demonstrated. We believe that MTF-UNet has great potential for further development in the field of medical image segmentation. The relevant code and data can be found at https://github.com/LCUDai/MTF-UNet.git.

A Generalizable Causal-Invariance-Driven Segmentation Model for Peripancreatic Vessels

Segmenting peripancreatic vessels in CT, including the superior mesenteric artery (SMA), the coeliac artery (CA), and the partial portal venous system (PPVS), is crucial for preoperative resectability analysis in pancreatic cancer. However, the clinical applicability of vessel segmentation methods is impeded by the low generalizability on multi-center data, mainly attributed to the wide variations in image appearance, namely the spurious correlation factor. Therefore, we propose a causal-invariance-driven generalizable segmentation model for peripancreatic vessels. It incorporates interventions at both image and feature levels to guide the model to capture causal information by enforcing consistency across datasets, thus enhancing the generalization performance. Specifically, firstly, a contrast-driven image intervention strategy is proposed to construct image-level interventions by generating images with various contrast-related appearances and seeking invariant causal features. Secondly, the feature intervention strategy is designed, where various patterns of feature bias across different centers are simulated to pursue invariant prediction. The proposed model achieved high DSC scores (79.69%, 82.62%, and 83.10%) for the three vessels on a cross-validation set containing 134 cases. Its generalizability was further confirmed on three independent test sets of 233 cases. Overall, the proposed method provides an accurate and generalizable segmentation model for peripancreatic vessels and offers a promising paradigm for increasing the generalizability of segmentation models from a causality perspective. Our source codes will be released at https://github.com/ SJTUBME-QianLab/PC_VesselSeg.

Semantic Segmentation of CT Liver Structures: A Systematic Review of Recent Trends and Bibliometric Analysis : Neural Network-based Methods for Liver Semantic Segmentation

The use of artificial intelligence (AI) in the segmentation of liver structures in medical images has become a popular research focus in the past half-decade. The performance of AI tools in screening for this task may vary widely and has been tested in the literature in various datasets. However, no scientometric report has provided a systematic overview of this scientific area. This article presents a systematic and bibliometric review of recent advances in neuronal network modeling approaches, mainly of deep learning, to outline the multiple research directions of the field in terms of algorithmic features. Therefore, a detailed systematic review of the most relevant publications addressing fully automatic semantic segmenting liver structures in Computed Tomography (CT) images in terms of algorithm modeling objective, performance benchmark, and model complexity is provided. The review suggests that fully automatic hybrid 2D and 3D networks are the top performers in the semantic segmentation of the liver. In the case of liver tumor and vasculature segmentation, fully automatic generative approaches perform best. However, the reported performance benchmark indicates that there is still much to be improved in segmenting such small structures in high-resolution abdominal CT scans.

Design of image segmentation model based on residual connection and feature fusion

With the development of deep learning technology, convolutional neural networks have made great progress in the field of image segmentation. However, for complex scenes and multi-scale target images, the existing technologies are still unable to achieve effective image segmentation. In view of this, an image segmentation model based on residual connection and feature fusion is proposed. The model makes comprehensive use of the deep feature extraction ability of residual connections and the multi-scale feature integration ability of feature fusion. In order to solve the problem of background complexity and information loss in traditional image segmentation, experiments were carried out on two publicly available data sets. The results showed that in the ISPRS Vaihingen dataset and the Caltech UCSD Birds200 dataset, when the model completed the 56th and 84th iterations, respectively, the average accuracy of FRes-MFDNN was the highest, which was 97.89% and 98.24%, respectively. In the ISPRS Vaihingen dataset and the Caltech UCSD Birds200 dataset, when the system model ran to 0.20s and 0.26s, the F1 value of the FRes-MFDNN method was the largest, and the F1 value approached 100% infinitely. The FRes-MFDNN segmented four images in the ISPRS Vaihingen dataset, and the segmentation accuracy of images 1, 2, 3 and 4 were 91.44%, 92.12%, 94.02% and 91.41%, respectively. In practical applications, the MSRF-Net method, LBN-AA-SPN method, ARG-Otsu method, and FRes-MFDNN were used to segment unlabeled bird images. The results showed that the FRes-MFDNN was more complete in details, and the overall effect was significantly better than the other three models. Meanwhile, in ordinary scene images, although there was a certain degree of noise and occlusion, the model still accurately recognized and segmented the main bird images. The results show that compared with the traditional model, after FRes-MFDNN segmentation, the completeness, detail, and spatial continuity of pixels have been significantly improved, making it more suitable for complex scenes.

Vascular liver segmentation: a narrative review on methods and new insights brought by artificial intelligence

Liver vessel segmentation from routinely performed medical imaging is a useful tool for diagnosis, treatment planning and delivery, and prognosis evaluation for many diseases, particularly liver cancer. A precise representation of liver anatomy is crucial to define the extent of the disease and, when suitable, the consequent resective or ablative procedure, in order to guarantee a radical treatment without sacrificing an excessive volume of healthy liver. Once mainly performed manually, with notable cost in terms of time and human energies, vessel segmentation is currently realized through the application of artificial intelligence (AI), which has gained increased interest and development of the field. Many different AI-driven models adopted for this aim have been described and can be grouped into different categories: thresholding methods, edge- and region-based methods, model-based methods, and machine learning models. The latter includes neural network and deep learning models that now represent the principal algorithms exploited for vessel segmentation. The present narrative review describes how liver vessel segmentation can be realized through AI models, with a summary of model results in terms of accuracy, and an overview on the future progress of this topic.

Hepatic and portal vein segmentation with dual-stream deep neural network

BACKGROUND

Liver lesions mainly occur inside the liver parenchyma, which are difficult to locate and have complicated relationships with essential vessels. Thus, preoperative planning is crucial for the resection of liver lesions. Accurate segmentation of the hepatic and portal veins (PVs) on computed tomography (CT) images is of great importance for preoperative planning. However, manually labeling the mask of vessels is laborious and time-consuming, and the labeling results of different clinicians are prone to inconsistencies. Hence, developing an automatic segmentation algorithm for hepatic and PVs on CT images has attracted the attention of researchers. Unfortunately, existing deep learning based automatic segmentation methods are prone to misclassifying peripheral vessels into wrong categories.

PURPOSE

This study aims to provide a fully automatic and robust semantic segmentation algorithm for hepatic and PVs, guiding subsequent preoperative planning. In addition, to address the deficiency of the public dataset for hepatic and PV segmentation, we revise the annotations of the Medical Segmentation Decathlon (MSD) hepatic vessel segmentation dataset and add the masks of the hepatic veins (HVs) and PVs.

METHODS

We proposed a structure with a dual-stream encoder combining convolution and Transformer block, named Dual-stream Hepatic Portal Vein segmentation Network, to extract local features and long-distance spatial information, thereby extracting anatomical information of hepatic and portal vein, avoiding misdivisions of adjacent peripheral vessels. Besides, a multi-scale feature fusion block based on dilated convolution is proposed to extract multi-scale features on expanded perception fields for local features, and a multi-level fusing attention module is introduced for efficient context information extraction. Paired t-test is conducted to evaluate the significant difference in dice between the proposed methods and the comparing methods.

RESULTS

Two datasets are constructed from the original MSD dataset. For each dataset, 50 cases are randomly selected for model evaluation in the scheme of 5-fold cross-validation. The results show that our method outperforms the state-of-the-art Convolutional Neural Network-based and transformer-based methods. Specifically, for the first dataset, our model reaches 0.815, 0.830, and 0.807 at overall dice, precision, and sensitivity. The dice of the hepatic and PVs are 0.835 and 0.796, which also exceed the numeric result of the comparing methods. Almost all the p-values of paired t-tests on the proposed approach and comparing approaches are smaller than 0.05. On the second dataset, the proposed algorithm achieves 0.749, 0.762, 0.726, 0.835, and 0.796 for overall dice, precision, sensitivity, dice for HV, and dice for PV, among which the first four numeric results exceed comparing methods.

CONCLUSIONS

The proposed method is effective in solving the problem of misclassifying interlaced peripheral veins for the HV and PV segmentation task and outperforming the comparing methods on the relabeled dataset.

Compensation of small data with large filters for accurate liver vessel segmentation from contrast-enhanced CT images

BACKGROUND

Segmenting liver vessels from contrast-enhanced computed tomography images is essential for diagnosing liver diseases, planning surgeries and delivering radiotherapy. Nevertheless, identifying vessels is a challenging task due to the tiny cross-sectional areas occupied by vessels, which has posed great challenges for vessel segmentation, such as limited features to be learned and difficult to construct high-quality as well as large-volume data.

METHODS

We present an approach that only requires a few labeled vessels but delivers significantly improved results. Our model starts with vessel enhancement by fading out liver intensity and generates candidate vessels by a classifier fed with a large number of image filters. Afterwards, the initial segmentation is refined using Markov random fields.

RESULTS

In experiments on the well-known dataset 3D-IRCADb, the averaged Dice coefficient is lifted to 0.63, and the mean sensitivity is increased to 0.71. These results are significantly better than those obtained from existing machine-learning approaches and comparable to those generated from deep-learning models.

CONCLUSION

Sophisticated integration of a large number of filters is able to pinpoint effective features from liver images that are sufficient to distinguish vessels from other liver tissues under a scarcity of large-volume labeled data. The study can shed light on medical image segmentation, especially for those without sufficient data.

SCAN: sequence-based context-aware association network for hepatic vessel segmentation

Accurate segmentation of hepatic vessel is significant for the surgeons to design the preoperative planning of liver surgery. In this paper, a sequence-based context-aware association network (SCAN) is designed for hepatic vessel segmentation, in which three schemes are incorporated to simultaneously extract the 2D features of hepatic vessels and capture the correlations between adjacent CT slices. The two schemes of slice-level attention module and graph association module are designed to bridge feature gaps between the encoder and the decoder in the low- and high-dimensional spaces. The region-edge constrained loss is designed to well optimize the proposed SCAN, which integrates cross-entropy loss, dice loss, and edge-constrained loss. Experimental results indicate that the proposed SCAN is superior to several existing deep learning frameworks, in terms of 0.845 DSC, 0.856 precision, 0.866 sensitivity, and 0.861 F1-score.

TaiChiNet: Negative-Positive Cross-Attention Network for Breast Lesion Segmentation in Ultrasound Images

Breast lesion segmentation in ultrasound images is essential for computer-aided breast-cancer diagnosis. To improve the segmentation performance, most approaches design sophisticated deep-learning models by mining the patterns of foreground lesions and normal backgrounds simultaneously or by unilaterally enhancing foreground lesions via various focal losses. However, the potential of normal backgrounds is underutilized, which could reduce false positives by compacting the feature representation of all normal backgrounds. From a novel viewpoint of bilateral enhancement, we propose a negative-positive cross-attention network to concentrate on normal backgrounds and foreground lesions, respectively. Derived from the complementing opposites of bipolarity in TaiChi, the network is denoted as TaiChiNet, which consists of the negative normal-background and positive foreground-lesion paths. To transmit the information across the two paths, a cross-attention module, a complementary MLP-head, and a complementary loss are built for deep-layer features, shallow-layer features, and mutual-learning supervision, separately. To the best of our knowledge, this is the first work to formulate breast lesion segmentation as a mutual supervision task from the foreground-lesion and normal-background views. Experimental results have demonstrated the effectiveness of TaiChiNet on two breast lesion segmentation datasets with a lightweight architecture. Furthermore, extensive experiments on the thyroid nodule segmentation and retinal optic cup/disc segmentation datasets indicate the application potential of TaiChiNet.

PEA-Net: A progressive edge information aggregation network for vessel segmentation

Automatic vessel segmentation is a critical area of research in medical image analysis, as it can greatly assist doctors in accurately and efficiently diagnosing vascular diseases. However, accurately extracting the complete vessel structure from images remains a challenge due to issues such as uneven contrast and background noise. Existing methods primarily focus on segmenting individual pixels and often fail to consider vessel features and morphology. As a result, these methods often produce fragmented results and misidentify vessel-like background noise, leading to missing and outlier points in the overall segmentation. To address these issues, this paper proposes a novel approach called the progressive edge information aggregation network for vessel segmentation (PEA-Net). The proposed method consists of several key components. First, a dual-stream receptive field encoder (DRE) is introduced to preserve fine structural features and mitigate false positive predictions caused by background noise. This is achieved by combining vessel morphological features obtained from different receptive field sizes. Second, a progressive complementary fusion (PCF) module is designed to enhance fine vessel detection and improve connectivity. This module complements the decoding path by combining features from previous iterations and the DRE, incorporating nonsalient information. Additionally, segmentation-edge decoupling enhancement (SDE) modules are employed as decoders to integrate upsampling features with nonsalient information provided by the PCF. This integration enhances both edge and segmentation information. The features in the skip connection and decoding path are iteratively updated to progressively aggregate fine structure information, thereby optimizing segmentation results and reducing topological disconnections. Experimental results on multiple datasets demonstrate that the proposed PEA-Net model and strategy achieve optimal performance in both pixel-level and topology-level metrics.

Optimization of table tennis target detection algorithm guided by multi-scale feature fusion of deep learning

This paper aims to propose a table tennis target detection (TD) method based on deep learning (DL) and multi-scale feature fusion (MFF) to improve the detection accuracy of the ball in table tennis competition, optimize the training process of athletes, and improve the technical level. In this paper, DL technology is used to improve the accuracy of table tennis TD through MFF guidance. Initially, based on the FAST Region-based Convolutional Neural Network (FAST R-CNN), the TD is carried out in the table tennis match. Then, through the method of MFF guidance, different levels of feature information are fused, which improves the accuracy of TD. Through the experimental verification on the test set, it is found that the mean Average Precision (mAP) value of the target detection algorithm (TDA) proposed here reaches 87.3%, which is obviously superior to other TDAs and has higher robustness. The DL TDA combined with the proposed MFF can be applied to various detection fields and can help the application of TD in real life.

Coronary artery segmentation in CCTA images based on multi-scale feature learning

BACKGROUND

Coronary artery segmentation is a prerequisite in computer-aided diagnosis of Coronary Artery Disease (CAD). However, segmentation of coronary arteries in Coronary Computed Tomography Angiography (CCTA) images faces several challenges. The current segmentation approaches are unable to effectively address these challenges and existing problems such as the need for manual interaction or low segmentation accuracy.

OBJECTIVE

A Multi-scale Feature Learning and Rectification (MFLR) network is proposed to tackle the challenges and achieve automatic and accurate segmentation of coronary arteries.

METHODS

The MFLR network introduces a multi-scale feature extraction module in the encoder to effectively capture contextual information under different receptive fields. In the decoder, a feature correction and fusion module is proposed, which employs high-level features containing multi-scale information to correct and guide low-level features, achieving fusion between the two-level features to further improve segmentation performance.

RESULTS

The MFLR network achieved the best performance on the dice similarity coefficient, Jaccard index, Recall, F1-score, and 95% Hausdorff distance, for both in-house and public datasets.

CONCLUSION

Experimental results demonstrate the superiority and good generalization ability of the MFLR approach. This study contributes to the accurate diagnosis and treatment of CAD, and it also informs other segmentation applications in medicine.

Progress and clinical translation in hepatocellular carcinoma of deep learning in hepatic vascular segmentation

This paper reviews the advancements in deep learning for hepatic vascular segmentation and its clinical implications in the holistic management of hepatocellular carcinoma (HCC). The key to the diagnosis and treatment of HCC lies in imaging examinations, with the challenge in liver surgery being the precise assessment of Hepatic vasculature. In this regard, deep learning methods, including convolutional neural networksamong various other approaches, have significantly improved accuracy and speed. The review synthesizes findings from 30 studies, covering aspects such as network architectures, applications, supervision techniques, evaluation metrics, and motivations. Furthermore, we also examine the challenges and future prospects of deep learning technologies in enhancing the comprehensive diagnosis and treatment of HCC, discussing anticipated breakthroughs that could transform patient management. By combining clinical needs with technological advancements, deep learning is expected to make greater breakthroughs in the field of hepatic vascular segmentation, thereby providing stronger support for the diagnosis and treatment of HCC.

Research on prognostic risk assessment model for acute ischemic stroke based on imaging and multidimensional data

Accurately assessing the prognostic outcomes of patients with acute ischemic stroke and adjusting treatment plans in a timely manner for those with poor prognosis is crucial for intervening in modifiable risk factors. However, there is still controversy regarding the correlation between imaging-based predictions of complications in acute ischemic stroke. To address this, we developed a cross-modal attention module for integrating multidimensional data, including clinical information, imaging features, treatment plans, prognosis, and complications, to achieve complementary advantages. The fused features preserve magnetic resonance imaging (MRI) characteristics while supplementing clinical relevant information, providing a more comprehensive and informative basis for clinical diagnosis and treatment. The proposed framework based on multidimensional data for activity of daily living (ADL) scoring in patients with acute ischemic stroke demonstrates higher accuracy compared to other state-of-the-art network models, and ablation experiments confirm the effectiveness of each module in the framework.

Specular highlight removal for endoscopic images using partial attention network

Objective.Endoscopic imaging is a visualization method widely used in minimally invasive surgery. However, owing to the strong reflection of the mucus layer on the organs, specular highlights often appear to degrade the imaging performance. Thus, it is necessary to develop an effective highlight removal method for endoscopic imaging.Approach.A specular highlight removal method using a partial attention network (PatNet) for endoscopic imaging is proposed to reduce the interference of bright light in endoscopic surgery. The method is designed as two procedures: highlight segmentation and endoscopic image inpainting. Image segmentation uses brightness threshold based on illumination compensation to divide the endoscopic image into the highlighted mask and the non-highlighted area. The image inpainting algorithm uses a partial convolution network that integrates an attention mechanism. A mask dataset with random hopping points is designed to simulate specular highlight in endoscopic imaging for network training. Through the filtering of masks, the method can focus on recovering defective pixels and preserving valid pixels as much as possible.Main results.The PatNet is compared with 3 highlight segmentation methods, 3 imaging inpainting methods and 5 highlight removal methods for effective analysis. Experimental results show that the proposed method provides better performance in terms of both perception and quantification. In addition, surgeons are invited to score the processing results for different highlight removal methods under realistic reflection conditions. The PatNet received the highest score of 4.18. Correspondingly, the kendall's W is 0.757 and the asymptotic significancep= 0.000 < 0.01, revealing that the subjective scores have good consistency and confidence.Significance.Generally, the method can realize irregular shape highlight reflection removal and image restoration close to the ground truth of endoscopic images. This method can improve the quality of endoscopic imaging for accurate image analysis.

A collaborative multi-task learning method for BI-RADS category 4 breast lesion segmentation and classification of MRI images

BACKGROUND AND OBJECTIVE

The diagnosis of BI-RADS category 4 breast lesion is difficult because its probability of malignancy ranges from 2% to 95%. For BI-RADS category 4 breast lesions, MRI is one of the prominent noninvasive imaging techniques. In this paper, we research computer algorithms to segment lesions and classify the benign or malignant lesions in MRI images. However, this task is challenging because the BI-RADS category 4 lesions are characterized by irregular shape, imbalanced class, and low contrast.

METHODS

We fully utilize the intrinsic correlation between segmentation and classification tasks, where accurate segmentation will yield accurate classification results, and classification results will promote better segmentation. Therefore, we propose a collaborative multi-task algorithm (CMTL-SC). Specifically, a preliminary segmentation subnet is designed to identify the boundaries, locations and segmentation masks of lesions; a classification subnet, which combines the information provided by the preliminary segmentation, is designed to achieve benign or malignant classification; a repartition segmentation subnet which aggregates the benign or malignant results, is designed to refine the lesion segment. The three subnets work cooperatively so that the CMTL-SC can identify the lesions better which solves the three challenges.

RESULTS AND CONCLUSION

We collect MRI data from 248 patients in the Second Hospital of Dalian Medical University. The results show that the lesion boundaries delineated by the CMTL-SC are close to the boundaries delineated by the physicians. Moreover, the CMTL-SC yields better results than the single-task and multi-task state-of-the-art algorithms. Therefore, CMTL-SC can help doctors make precise diagnoses and refine treatments for patients.

PMJAF-Net: Pyramidal multi-scale joint attention and adaptive fusion network for explainable skin lesion segmentation

Traditional convolutional neural networks have achieved remarkable success in skin lesion segmentation. However, the successive pooling operations and convolutional spans reduce the feature resolution and hinder the dense prediction for spatial information, resulting in blurred boundaries, low accuracy and poor interpretability for irregular lesion segmentation under low contrast. To solve the above issues, a pyramidal multi-scale joint attention and adaptive fusion network for explainable (PMJAF-Net) skin lesion segmentation is proposed. Firstly, an adaptive spatial attention module is designed to establish the long-term correlation between pixels, enrich the global and local contextual information, and refine the detailed features. Subsequently, an efficient pyramidal multi-scale channel attention module is proposed to capture the multi-scale information and edge features by using the pyramidal module. Meanwhile, a channel attention module is devised to establish the long-term correlation between channels and highlight the most related feature channels to capture the multi-scale key information on each channel. Thereafter, a multi-scale adaptive fusion attention module is put forward to efficiently fuse the scale features at different decoding stages. Finally, a novel hybrid loss function based on region salient features and boundary quality is presented to guide the network to learn from map-level, patch-level and pixel-level and to accurately predict the lesion regions with clear boundaries. In addition, visualizing attention weight maps are utilized to visually enhance the interpretability of our proposed model. Comprehensive experiments are conducted on four public skin lesion datasets, and the results demonstrate that the proposed network outperforms the state-of-the-art methods, with the segmentation assessment evaluation metrics Dice, JI, and ACC improved to 92.65%, 87.86% and 96.26%, respectively.

Global Transformer and Dual Local Attention Network via Deep-Shallow Hierarchical Feature Fusion for Retinal Vessel Segmentation

Clinically, retinal vessel segmentation is a significant step in the diagnosis of fundus diseases. However, recent methods generally neglect the difference of semantic information between deep and shallow features, which fail to capture the global and local characterizations in fundus images simultaneously, resulting in the limited segmentation performance for fine vessels. In this article, a global transformer (GT) and dual local attention (DLA) network via deep-shallow hierarchical feature fusion (GT-DLA-dsHFF) are investigated to solve the above limitations. First, the GT is developed to integrate the global information in the retinal image, which effectively captures the long-distance dependence between pixels, alleviating the discontinuity of blood vessels in the segmentation results. Second, DLA, which is constructed using dilated convolutions with varied dilation rates, unsupervised edge detection, and squeeze-excitation block, is proposed to extract local vessel information, consolidating the edge details in the segmentation result. Finally, a novel deep-shallow hierarchical feature fusion (dsHFF) algorithm is studied to fuse the features in different scales in the deep learning framework, respectively, which can mitigate the attenuation of valid information in the process of feature fusion. We verified the GT-DLA-dsHFF on four typical fundus image datasets. The experimental results demonstrate our GT-DLA-dsHFF achieves superior performance against the current methods and detailed discussions verify the efficacy of the proposed three modules. Segmentation results of diseased images show the robustness of our proposed GT-DLA-dsHFF. Implementation codes will be available on https://github.com/YangLibuaa/GT-DLA-dsHFF.

EIU-Net: Enhanced feature extraction and improved skip connections in U-Net for skin lesion segmentation

Skin lesion segmentation is a computer-aided diagnosis method for quantitative analysis of melanoma that can improve efficiency and accuracy. Although many methods based on U-Net have achieved tremendous success, they still cannot handle challenging tasks well due to weak feature extraction. In response to skin lesion segmentation, a novel method called EIU-Net is proposed to tackle the challenging task. To capture the local and global contextual information, we employ inverted residual blocks and an efficient pyramid squeeze attention (EPSA) block as the main encoders at different stages, while atrous spatial pyramid pooling (ASPP) is utilized after the last encoder and the soft-pool method is introduced for downsampling. Also, we propose a novel method named multi-layer fusion (MLF) module to effectively fuse the feature distributions and capture significant boundary information of skin lesions in different encoders to improve the performance of the network. Furthermore, a reshaped decoders fusion module is used to obtain multi-scale information by fusing feature maps of different decoders to improve the final results of skin lesion segmentation. To validate the performance of our proposed network, we compare it with other methods on four public datasets, including the ISIC 2016, ISIC 2017, ISIC 2018, and PH2 datasets. And the main metric Dice scores achieved by our proposed EIU-Net are 0.919, 0.855, 0.902, and 0.916 on the four datasets, respectively, outperforming other methods. Ablation experiments also demonstrate the effectiveness of the main modules in our proposed network. Our code is available at https://github.com/AwebNoob/EIU-Net.

Affinity Feature Strengthening for Accurate, Complete and Robust Vessel Segmentation

Vessel segmentation is crucial in many medical image applications, such as detecting coronary stenoses, retinal vessel diseases and brain aneurysms. However, achieving high pixel-wise accuracy, complete topology structure and robustness to various contrast variations are critical and challenging, and most existing methods focus only on achieving one or two of these aspects. In this paper, we present a novel approach, the affinity feature strengthening network (AFN), which jointly models geometry and refines pixel-wise segmentation features using a contrast-insensitive, multiscale affinity approach. Specifically, we compute a multiscale affinity field for each pixel, capturing its semantic relationships with neighboring pixels in the predicted mask image. This field represents the local geometry of vessel segments of different sizes, allowing us to learn spatial- and scale-aware adaptive weights to strengthen vessel features. We evaluate our AFN on four different types of vascular datasets: X-ray angiography coronary vessel dataset (XCAD), portal vein dataset (PV), digital subtraction angiography cerebrovascular vessel dataset (DSA) and retinal vessel dataset (DRIVE). Extensive experimental results demonstrate that our AFN outperforms the state-of-the-art methods in terms of both higher accuracy and topological metrics, while also being more robust to various contrast changes.

Hepatic vessel segmentation based on 3D swin-transformer with inductive biased multi-head self-attention

PURPOSE

Segmentation of liver vessels from CT images is indispensable prior to surgical planning and aroused a broad range of interest in the medical image analysis community. Due to the complex structure and low-contrast background, automatic liver vessel segmentation remains particularly challenging. Most of the related researches adopt FCN, U-net, and V-net variants as a backbone. However, these methods mainly focus on capturing multi-scale local features which may produce misclassified voxels due to the convolutional operator's limited locality reception field.

METHODS

We propose a robust end-to-end vessel segmentation network called Inductive BIased Multi-Head Attention Vessel Net(IBIMHAV-Net) by expanding swin transformer to 3D and employing an effective combination of convolution and self-attention. In practice, we introduce voxel-wise embedding rather than patch-wise embedding to locate precise liver vessel voxels and adopt multi-scale convolutional operators to gain local spatial information. On the other hand, we propose the inductive biased multi-head self-attention which learns inductively biased relative positional embedding from initialized absolute position embedding. Based on this, we can gain more reliable queries and key matrices.

RESULTS

We conducted experiments on the 3DIRCADb dataset. The average dice and sensitivity of the four tested cases were 74.8[Formula: see text] and 77.5[Formula: see text], which exceed the results of existing deep learning methods and improved graph cuts method. The Branches Detected(BD)/Tree-length Detected(TD) indexes also proved the global/local feature capture ability better than other methods.

CONCLUSION

The proposed model IBIMHAV-Net provides an automatic, accurate 3D liver vessel segmentation with an interleaved architecture that better utilizes both global and local spatial features in CT volumes. It can be further extended for other clinical data.

DHT-Net: Dynamic Hierarchical Transformer Network for Liver and Tumor Segmentation

Automatic segmentation of liver tumors is crucial to assist radiologists in clinical diagnosis. While various deep learningbased algorithms have been proposed, such as U-Net and its variants, the inability to explicitly model long-range dependencies in CNN limits the extraction of complex tumor features. Some researchers have applied Transformer-based 3D networks to analyze medical images. However, the previous methods focus on modeling the local information (eg. edge) or global information (eg. morphology) with fixed network weights. To learn and extract complex tumor features of varied tumor size, location, and morphology for more accurate segmentation, we propose a Dynamic Hierarchical Transformer Network, named DHT-Net. The DHT-Net mainly contains a Dynamic Hierarchical Transformer (DHTrans) structure and an Edge Aggregation Block (EAB). The DHTrans first automatically senses the tumor location by Dynamic Adaptive Convolution, which employs hierarchical operations with the different receptive field sizes to learn the features of various tumors, thus enhancing the semantic representation ability of tumor features. Then, to adequately capture the irregular morphological features in the tumor region, DHTrans aggregates global and local texture information in a complementary manner. In addition, we introduce the EAB to extract detailed edge features in the shallow fine-grained details of the network, which provides sharp boundaries of liver and tumor regions. We evaluate DHT-Net on two challenging public datasets, LiTS and 3DIRCADb. The proposed method has shown superior liver and tumor segmentation performance compared to several state-of-the-art 2D, 3D, and 2.5D hybrid models.

All answers are in the images: A review of deep learning for cerebrovascular segmentation

Cerebrovascular imaging is a common examination. Its accurate cerebrovascular segmentation become an important auxiliary method for the diagnosis and treatment of cerebrovascular diseases, which has received extensive attention from researchers. Deep learning is a heuristic method that encourages researchers to derive answers from the images by driving datasets. With the continuous development of datasets and deep learning theory, it has achieved important success for cerebrovascular segmentation. Detailed survey is an important reference for researchers. To comprehensively analyze the newest cerebrovascular segmentation, we have organized and discussed researches centered on deep learning. This survey comprehensively reviews deep learning for cerebrovascular segmentation since 2015, it mainly includes sliding window based models, U-Net based models, other CNNs based models, small-sample based models, semi-supervised or unsupervised models, fusion based models, Transformer based models, and graphics based models. We organize the structures, improvement, and important parameters of these models, as well as analyze development trends and quantitative assessment. Finally, we have discussed the challenges and opportunities of possible research directions, hoping that our survey can provide researchers with convenient reference.

A novel multi-attention, multi-scale 3D deep network for coronary artery segmentation

Automatic segmentation of coronary arteries provides vital assistance to enable accurate and efficient diagnosis and evaluation of coronary artery disease (CAD). However, the task of coronary artery segmentation (CAS) remains highly challenging due to the large-scale variations exhibited by coronary arteries, their complicated anatomical structures and morphologies, as well as the low contrast between vessels and their background. To comprehensively tackle these challenges, we propose a novel multi-attention, multi-scale 3D deep network for CAS, which we call CAS-Net. Specifically, we first propose an attention-guided feature fusion (AGFF) module to efficiently fuse adjacent hierarchical features in the encoding and decoding stages to capture more effectively latent semantic information. Then, we propose a scale-aware feature enhancement (SAFE) module, aiming to dynamically adjust the receptive fields to extract more expressive features effectively, thereby enhancing the feature representation capability of the network. Furthermore, we employ the multi-scale feature aggregation (MSFA) module to learn a more distinctive semantic representation for refining the vessel maps. In addition, considering that the limited training data annotated with a quality golden standard are also a significant factor restricting the development of CAS, we construct a new dataset containing 119 cases consisting of coronary computed tomographic angiography (CCTA) volumes and annotated coronary arteries. Extensive experiments on our self-collected dataset and three publicly available datasets demonstrate that the proposed method has good segmentation performance and generalization ability, outperforming multiple state-of-the-art algorithms on various metrics. Compared with U-Net3D, the proposed method significantly improves the Dice similarity coefficient (DSC) by at least 4% on each dataset, due to the synergistic effect among the three core modules, AGFF, SAFE, and MSFA. Our implementation is released at https://github.com/Cassie-CV/CAS-Net.

TP-Net: Two-Path Network for Retinal Vessel Segmentation

Refined and automatic retinal vessel segmentation is crucial for computer-aided early diagnosis of retinopathy. However, existing methods often suffer from mis-segmentation when dealing with thin and low-contrast vessels. In this paper, a two-path retinal vessel segmentation network is proposed, namely TP-Net, which consists of three core parts, i.e., main-path, sub-path, and multi-scale feature aggregation module (MFAM). Main-path is to detect the trunk area of the retinal vessels, and the sub-path to effectively capture edge information of the retinal vessels. The prediction results of the two paths are combined by MFAM, obtaining refined segmentation of retinal vessels. In the main-path, a three-layer lightweight backbone network is elaborately designed according to the characteristics of retinal vessels, and then a global feature selection mechanism (GFSM) is proposed, which can autonomously select features that are more important for the segmentation task from the features at different layers of the network, thereby, enhancing the segmentation capability for low-contrast vessels. In the sub-path, an edge feature extraction method and an edge loss function are proposed, which can enhance the ability of the network to capture edge information and reduce the mis-segmentation of thin vessels. Finally, MFAM is proposed to fuse the prediction results of main-path and sub-path, which can remove background noises while preserving edge details, and thus, obtaining refined segmentation of retinal vessels. The proposed TP-Net has been evaluated on three public retinal vessel datasets, namely DRIVE, STARE, and CHASE DB1. The experimental results show that the TP-Net achieved a superior performance and generalization ability with fewer model parameters compared with the state-of-the-art methods.

Learning multi-view and centerline topology connectivity information for pulmonary artery-vein separation

BACKGROUND

Automatic pulmonary artery-vein separation has considerable importance in the diagnosis and treatment of lung diseases. However, insufficient connectivity and spatial inconsistency have always been the problems of artery-vein separation.

METHODS

A novel automatic method for artery-vein separation in CT images is presented in this work. Specifically, a multi-scale information aggregated network (MSIA-Net) including multi-scale fusion blocks and deep supervision, is proposed to learn the features of artery-vein and aggregate additional semantic information, respectively. The proposed method integrates nine MSIA-Net models for artery-vein separation, vessel segmentation, and centerline separation tasks along with axial, coronal, and sagittal multi-view slices. First, the preliminary artery-vein separation results are obtained by the proposed multi-view fusion strategy (MVFS). Then, centerline correction algorithm (CCA) is used to correct the preliminary results of artery-vein separation by the centerline separation results. Finally, the vessel segmentation results are utilized to reconstruct the artery-vein morphology. In addition, weighted cross-entropy and dice loss are employed to solve the class imbalance problem.

RESULTS

We constructed 50 manually labeled contrast-enhanced computed CT scans for five-fold cross-validation, and experimental results demonstrated that our method achieves superior segmentation performance of 97.7%, 85.1%, and 84.9% on ACC, Pre, and DSC, respectively. Additionally, a series of ablation studies demonstrate the effectiveness of the proposed components.

CONCLUSION

The proposed method can effectively solve the problem of insufficient vascular connectivity and correct the spatial inconsistency of artery-vein.

A venipuncture robot with decoupled position and attitude guided by near-infrared vision and force feedback

BACKGROUND

This study aims to develop a venipuncture robot to replace manual venipuncture to ease the heavy workload, lower the risk of 2019-nCoV infection, and boost venipuncture success rates.

METHOD

The robot is designed with decoupled position and attitude. It consists of a 3-degree-of-freedom positioning manipulator to locate the needle and a 3-degree-of-freedom end-effector that is always vertical to adjust the yaw and pitch angles of the needle. The near-infrared vision and laser sensors obtain the three-dimensional information of puncture positions, while the change in force detects the state feedback of punctures.

RESULTS

The experimental results demonstrate that the venipuncture robot has a compact design, flexible motion, high positioning accuracy and repeatability (0.11 and 0.04 mm), and a high success rate when puncturing the phantom.

CONCLUSION

This paper presents a decoupled position and attitude venipuncture robot guided by near-infrared vision and force feedback to replace manual venipuncture. The robot is compact, dexterous, and accurate, which helps to improve the success rate of venipuncture, and it is expected to achieve fully automatic venipuncture in the future.

A deep residual attention-based U-Net with a biplane joint method for liver segmentation from CT scans

Liver tumours are diseases with high morbidity and high deterioration probabilities, and accurate liver area segmentation from computed tomography (CT) scans is a prerequisite for quick tumour diagnosis. While 2D network segmentation methods can perform segmentation with lower device performance requirements, they often discard the rich 3D spatial information contained in CT scans, limiting their segmentation accuracy. Hence, a deep residual attention-based U-shaped network (DRAUNet) with a biplane joint method for liver segmentation is proposed in this paper, where the biplane joint method introduces coronal CT slices to assist the transverse slices with segmentation, incorporating more 3D spatial information into the segmentation results to improve the segmentation performance of the network. Additionally, a novel deep residual block (DR block) and dual-effect attention module (DAM) are introduced in DRAUNet, where the DR block has deeper layers and two shortcut paths. The DAM efficiently combines the correlations of feature channels and the spatial locations of feature maps. The DRAUNet with the biplane joint method is tested on three datasets, Liver Tumour Segmentation (LiTS), 3D Image Reconstruction for Comparison of Algorithms Database (3DIRCADb), and Segmentation of the Liver Competition 2007 (Sliver07), and it achieves 97.3%, 97.4%, and 96.9% Dice similarity coefficients (DSCs) for liver segmentation, respectively, outperforming most state-of-the-art networks; this strongly demonstrates the segmentation performance of DRAUNet and the ability of the biplane joint method to obtain 3D spatial information from 3D images.

P-ResUnet: Segmentation of brain tissue with Purified Residual Unet

Brain tissue of Magnetic Resonance Imaging is precisely segmented and quantified, which aids in the diagnosis of neurological diseases such as epilepsy, Alzheimer's, and multiple sclerosis. Recently, UNet-like architectures are widely used for medical image segmentation, which achieved promising performance by using the skip connection to fuse the low-level and high-level information. However, In the process of integrating the low-level and high-level information, the non-object information (noise) will be added, which reduces the accuracy of medical image segmentation. Likewise, the same problem also exists in the residual unit. Since the output and input of the residual unit are fused, the non-object information (noise) of the input of the residual unit will be in the integration. To address this challenging problem, in this paper we propose a Purified Residual U-net for the segmentation of brain tissue. This model encodes the image to obtain deep semantic information and purifies the information of low-level features and the residual unit from the image, and acquires the result through a decoder at last. We use the Dilated Pyramid Separate Block (DPSB) as the first block to purify the features for each layer in the encoder without the first layer, which expands the receptive field of the convolution kernel with only a few parameters added. In the first layer, we have explored the best performance achieved with DPB. We find the most non-object information (noise) in the initial image, so it is good for the accuracy to exchange the information to the max degree. We have conducted experiments with the widely used IBSR-18 dataset composed of T-1 weighted MRI volumes from 18 subjects. The results show that compared with some of the cutting-edge methods, our method enhances segmentation performance with the mean dice score reaching 91.093% and the mean Hausdorff distance decreasing to 3.2606.

Hepatic Vein and Arterial Vessel Segmentation in Liver Tumor Patients

Preoperative observation of liver status in patients with liver tumors by abdominal Computed Tomography (CT) imaging is one of the essential references for formulating surgical plans. Preoperative vessel segmentation in the patient's liver region has become an increasingly important and challenging problem. Almost all existing methods first segment arterial and venous vessels on CT in the arterial and venous phases, respectively. Then, the two are directly registered to complete the reconstruction of the vascular system, ignoring the displacement and deformation of blood vessels caused by changes in body position and respiration in the two phases. We propose an unsupervised domain-adaptive two-stage vessel segmentation framework for simultaneous fine segmentation of arterial and venous vessels on venous phase CT. Specifically, we first achieve domain adaptation for arterial and venous phase CT using a modified cycle-consistent adversarial network. The newly added discriminator can improve the ability to generate and discriminate tiny blood vessels, making the domain-adaptive network more robust. The second-stage supervised training of arterial vessels was then performed on the translated arterial phase CT. In this process, we propose an orthogonal depth projection loss function to enhance the representation ability of the 3D U-shape segmentation network for the geometric information of the vessel model. The segmented venous vessels were also performed on venous phase CT in the second stage. Finally, we invited professional doctors to annotate arterial and venous vessels on the venous phase CT of the test set. The experimental results show that the segmentation accuracy of arterial and venous vessels on venous phase CT is 0.8454 and 0.8087, respectively. Our proposed framework can simultaneously achieve supervised segmentation of venous vessels and unsupervised segmentation of arterial vessels on venous phase CT. Our approach can be extended to other fields of medical segmentation, such as unsupervised domain adaptive segmentation of liver tumors at different CT phases, to facilitate the development of the community.

Dual Encoder-Based Dynamic-Channel Graph Convolutional Network With Edge Enhancement for Retinal Vessel Segmentation

Retinal vessel segmentation with deep learning technology is a crucial auxiliary method for clinicians to diagnose fundus diseases. However, the deep learning approaches inevitably lose the edge information, which contains spatial features of vessels while performing down-sampling, leading to the limited segmentation performance of fine blood vessels. Furthermore, the existing methods ignore the dynamic topological correlations among feature maps in the deep learning framework, resulting in the inefficient capture of the channel characterization. To address these limitations, we propose a novel dual encoder-based dynamic-channel graph convolutional network with edge enhancement (DE-DCGCN-EE) for retinal vessel segmentation. Specifically, we first design an edge detection-based dual encoder to preserve the edge of vessels in down-sampling. Secondly, we investigate a dynamic-channel graph convolutional network to map the image channels to the topological space and synthesize the features of each channel on the topological map, which solves the limitation of insufficient channel information utilization. Finally, we study an edge enhancement block, aiming to fuse the edge and spatial features in the dual encoder, which is beneficial to improve the accuracy of fine blood vessel segmentation. Competitive experimental results on five retinal image datasets validate the efficacy of the proposed DE-DCGCN-EE, which achieves more remarkable segmentation results against the other state-of-the-art methods, indicating its potential clinical application.

Trans-arterial positive ICG staining-guided laparoscopic liver watershed resection for hepatocellular carcinoma

Introduction

Anatomical liver resection is the optimal treatment for patients with resectable hepatocellular carcinoma (HCC). Laparoscopic Couinaud liver segment resection could be performed easily as liver segments could be stained by ultrasound-guided indocyanine green (ICG) injection into the corresponding segment portal vein. Several smaller liver anatomical units (liver watersheds) have been identified (such as S8v, S8d, S4a, and S4b). However, since portal veins of liver watersheds are too thin to be identified under ultrasound, the boundaries of these liver watersheds could not be stained intraoperatively, making laparoscopic resection of these liver watersheds demanding. Digital subtraction angiography (DSA) could identify arteries of liver watersheds with a diameter of less than 2 mm. Yet, its usage for liver watershed staining has not been explored so far.

Purpose

The aim of this study is to explore the possibility of positive liver watershed staining via trans-arterial ICG injection under DSA examination for navigating laparoscopic watershed-oriented hepatic resection.

Methods

We describe, in a step-by-step approach, the application of trans-arterial ICG injection to stain aimed liver watershed during laparoscopic anatomical hepatectomy. The efficiency and safety of the technique are illustrated and discussed in comparison with the laparoscopic anatomical liver resection via ultrasound-guided liver segment staining.

Results

Eight of 10 HCC patients received successful trans-arterial liver watershed staining. The success rate of the trans-artery staining approach was 80%, higher than that of the ultrasound-guided portal vein staining approach (60%). Longer surgical duration was found in patients who underwent the trans-artery staining approach (305.3 ± 23.2 min vs. 268.4 ± 34.7 min in patients who underwent the ultrasound-guided portal vein staining approach, p = 0.004). No significant difference was found in major morbidity, reoperation rate, hospital stay duration, and 30-day and 90-day mortality between the 2 groups.

Conclusions

Trans-arterial ICG staining is safe and feasible for staining the aimed liver watershed, navigating watershed-oriented hepatic resection under fluorescence laparoscopy for surgeons.

Multi-perspective label based deep learning framework for cerebral vasculature segmentation in whole-brain fluorescence images

The popularity of fluorescent labelling and mesoscopic optical imaging techniques enable the acquisition of whole mammalian brain vasculature images at capillary resolution. Segmentation of the cerebrovascular network is essential for analyzing the cerebrovascular structure and revealing the pathogenesis of brain diseases. Existing deep learning methods use a single type of annotated labels with the same pixel weight to train the neural network and segment vessels. Due to the variation in the shape, density and brightness of vessels in whole-brain fluorescence images, it is difficult for a neural network trained with a single type of label to segment all vessels accurately. To address this problem, we proposed a deep learning cerebral vasculature segmentation framework based on multi-perspective labels. First, the pixels in the central region of thick vessels and the skeleton region of vessels were extracted separately using morphological operations based on the binary annotated labels to generate two different labels. Then, we designed a three-stage 3D convolutional neural network containing three sub-networks, namely thick-vessel enhancement network, vessel skeleton enhancement network and multi-channel fusion segmentation network. The first two sub-networks were trained by the two labels generated in the previous step, respectively, and pre-segmented the vessels. The third sub-network was responsible for fusing the pre-segmented results to precisely segment the vessels. We validated our method on two mouse cerebral vascular datasets generated by different fluorescence imaging modalities. The results showed that our method outperforms the state-of-the-art methods, and the proposed method can be applied to segment the vasculature on large-scale volumes.

A Machine Learning Method for Automated In Vivo Transparent Vessel Segmentation and Identification Based on Blood Flow Characteristics

In vivo transparent vessel segmentation is important to life science research. However, this task remains very challenging because of the fuzzy edges and the barely noticeable tubular characteristics of vessels under a light microscope. In this paper, we present a new machine learning method based on blood flow characteristics to segment the global vascular structure in vivo. Specifically, the videos of blood flow in transparent vessels are used as input. We use the machine learning classifier to classify the vessel pixels through the motion features extracted from moving red blood cells and achieve vessel segmentation based on a region-growing algorithm. Moreover, we utilize the moving characteristics of blood flow to distinguish between the types of vessels, including arteries, veins, and capillaries. In the experiments, we evaluate the performance of our method on videos of zebrafish embryos. The experimental results indicate the high accuracy of vessel segmentation, with an average accuracy of 97.98%, which is much more superior than other segmentation or motion-detection algorithms. Our method has good robustness when applied to input videos with various time resolutions, with a minimum of 3.125 fps.

Techniques and Algorithms for Hepatic Vessel Skeletonization in Medical Images: A Survey

Hepatic vessel skeletonization serves as an important means of hepatic vascular analysis and vessel segmentation. This paper presents a survey of techniques and algorithms for hepatic vessel skeletonization in medical images. We summarized the latest developments and classical approaches in this field. These methods are classified into five categories according to their methodological characteristics. The overview and brief assessment of each category are provided in the corresponding chapters, respectively. We provide a comprehensive summary among the cited publications, image modalities and datasets from various aspects, which hope to reveal the pros and cons of every method, summarize its achievements and discuss the challenges and future trends.

Adaptive brightness fusion method for intraoperative near-infrared fluorescence and visible images

An adaptive brightness fusion method (ABFM) for near-infrared fluorescence imaging is proposed to adapt to different lighting conditions and make the equipment operation more convenient in clinical applications. The ABFM is designed based on the network structure of Attention Unet, which is an image segmentation technique. Experimental results show that ABFM has the function of adaptive brightness adjustment and has better fusion performance in terms of both perception and quantification. Generally, the proposed method can realize an adaptive brightness fusion of fluorescence and visible images to enhance the usability of fluorescence imaging technology during surgery.

Use of the volume-averaged Murray's deviation method for the characterization of branching geometry in liver fibrosis: a preliminary study on vascular circulation

Background

Vascular changes in liver fibrosis can result in increased intrahepatic vascular resistance and impaired blood circulation. This can hinder the recovery from fibrosis and may eventually lead to portal hypertension, a major cirrhosis complication. This report proposed a volume-averaged Murray's deviation method to characterize intrahepatic circulation in the liver during fibrosis and its subsequent regression via X-ray phase-contrast computed tomography (PCCT).

Methods

Liver fibrosis was induced in 24 Sprague-Dawley rats by exposure to carbon tetrachloride (CCl₄) for up to 10 weeks, after which, spontaneous regression commenced and continued until week 30. High-resolution three-dimensional (3D) imaging of the livers was performed with PCCT. The values of Murray's deviation based on the volume-averaged and the conventional diameter-based methods were compared. After that, the intrahepatic circulation at different stages of fibrosis was evaluated using the volume-averaged method. The increase in collagen during liver fibrosis was assessed by pathological analyses.

Results

A comparison of the 2 methods showed that with an increase in the number of diameter measurements, the value of Murrary's deviation obtained using the diameter-based method gradually approaches those of the volume-averaged method, with minimal variations. The value of Murray's deviation increased with the development of fibrosis. After reversal, the value rapidly decreased and approached that of the normal state in both the main branches (1.05±0.17, 1.17±0.21, 1.34±0.18, and 1.17±0.19 in the normal, moderate, severe, and regressive groups, respectively; P<0.05 between the severe group and other groups) and the small branches (1.05±0.09, 1.42±0.48, 1.79±0.57, and 1.18±0.28 in the normal, moderate, severe, and regressive group, respectively; P<0.05 between adjacent groups). An analysis of Murray's deviation and the pathological results showed that the vascular circulation in this disease model was consistent with the progression and recovery from fibrosis.

Conclusions

This study showed the validity of the volume-averaged method for calculating Murray's deviation and demonstrated that it could accurately evaluate the blood circulation state of the liver during fibrosis and its subsequent regression. Thus, the volume-averaged method of calculating Murray's deviation may be an objective and valuable staging criterion to evaluate intrahepatic circulation during liver fibrosis.

Multi-channel convolutional neural network architectures for thyroid cancer detection

Early detection of malignant thyroid nodules leading to patient-specific treatments can reduce morbidity and mortality rates. Currently, thyroid specialists use medical images to diagnose then follow the treatment protocols, which have limitations due to unreliable human false-positive diagnostic rates. With the emergence of deep learning, advances in computer-aided diagnosis techniques have yielded promising earlier detection and prediction accuracy; however, clinicians' adoption is far lacking. The present study adopts Xception neural network as the base structure and designs a practical framework, which comprises three adaptable multi-channel architectures that were positively evaluated using real-world data sets. The proposed architectures outperform existing statistical and machine learning techniques and reached a diagnostic accuracy rate of 0.989 with ultrasound images and 0.975 with computed tomography scans through the single input dual-channel architecture. Moreover, the patient-specific design was implemented for thyroid cancer detection and has obtained an accuracy of 0.95 for double inputs dual-channel architecture and 0.94 for four-channel architecture. Our evaluation suggests that ultrasound images and computed tomography (CT) scans yield comparable diagnostic results through computer-aided diagnosis applications. With ultrasound images obtained slightly higher results, CT, on the other hand, can achieve the patient-specific diagnostic design. Besides, with the proposed framework, clinicians can select the best fitting architecture when making decisions regarding a thyroid cancer diagnosis. The proposed framework also incorporates interpretable results as evidence, which potentially improves clinicians' trust and hence their adoption of the computer-aided diagnosis techniques proposed with increased efficiency and accuracy.

3D Graph-Connectivity Constrained Network for Hepatic Vessel Segmentation

Segmentation of hepatic vessels from 3D CT images is necessary for accurate diagnosis and preoper-ative planning for liver cancer. However, due to the low contrast and high noises of CT images, automatic hepatic vessel segmentation is a challenging task. Hepatic vessels are connected branches containing thick and thin blood vessels, showing an important structural characteristic or a prior: the connectivity of blood vessels. However, this is rarely applied in existing methods. In this paper, we segment hepatic vessels from 3D CT images by utilizing the connectivity prior. To this end, a graph neural network (GNN) used to describe the connectivity prior of hepatic vessels is integrated into a general convolutional neu-ral network (CNN). Specifically, a graph attention network (GAT) is first used to model the graphical connectivity information of hepatic vessels, which can be trained with the vascular connectivity graph constructed directly from the ground truths. Second, the GAT is integrated with a lightweight 3D U-Net by an efficient mechanism called the plug-in mode, in which the GAT is incorporated into the U-Net as a multi-task branch and is only used to supervise the training procedure of the U-Net with the connectivity prior. The GAT will not be used in the inference stage, and thus will not increase the hardware and time costs of the inference stage compared with the U-Net. Therefore, hepatic vessel segmentation can be well improved in an efficient mode. Extensive experiments on two public datasets show that the proposed method is superior to related works in accuracy and connectivity of hepatic vessel segmentation.

Multilevel Deep-Aggregated Boosted Network to Recognize COVID-19 Infection from Large-Scale Heterogeneous Radiographic Data

In the present epidemic of the coronavirus disease 2019 (COVID-19), radiological imaging modalities, such as X-ray and computed tomography (CT), have been identified as effective diagnostic tools. However, the subjective assessment of radiographic examination is a time-consuming task and demands expert radiologists. Recent advancements in artificial intelligence have enhanced the diagnostic power of computer-aided diagnosis (CAD) tools and assisted medical specialists in making efficient diagnostic decisions. In this work, we propose an optimal multilevel deep-aggregated boosted network to recognize COVID-19 infection from heterogeneous radiographic data, including X-ray and CT images. Our method leverages multilevel deep-aggregated features and multistage training via a mutually beneficial approach to maximize the overall CAD performance. To improve the interpretation of CAD predictions, these multilevel deep features are visualized as additional outputs that can assist radiologists in validating the CAD results. A total of six publicly available datasets were fused to build a single large-scale heterogeneous radiographic collection that was used to analyze the performance of the proposed technique and other baseline methods. To preserve generality of our method, we selected different patient data for training, validation, and testing, and consequently, the data of same patient were not included in training, validation, and testing subsets. In addition, fivefold cross-validation was performed in all the experiments for a fair evaluation. Our method exhibits promising performance values of 95.38%, 95.57%, 92.53%, 98.14%, 93.16%, and 98.55% in terms of average accuracy, F-measure, specificity, sensitivity, precision, and area under the curve, respectively and outperforms various state-of-the-art methods.

Accurate Tumor Segmentation via Octave Convolution Neural Network

Three-dimensional (3D) liver tumor segmentation from Computed Tomography (CT) images is a prerequisite for computer-aided diagnosis, treatment planning, and monitoring of liver cancer. Despite many years of research, 3D liver tumor segmentation remains a challenging task. In this paper, we propose an effective and efficient method for tumor segmentation in liver CT images using encoder-decoder based octave convolution networks. Compared with other convolution networks utilizing standard convolution for feature extraction, the proposed method utilizes octave convolutions for learning multiple-spatial-frequency features, thus can better capture tumors with varying sizes and shapes. The proposed network takes advantage of a fully convolutional architecture which performs efficient end-to-end learning and inference. More importantly, we introduce a deep supervision mechanism during the learning process to combat potential optimization difficulties, and thus the model can acquire a much faster convergence rate and more powerful discrimination capability. Finally, we integrate octave convolutions into the encoder-decoder architecture of UNet, which can generate high resolution tumor segmentation in one single forward feeding without post-processing steps. Both architectures are trained on a subset of the LiTS (Liver Tumor Segmentation) Challenge. The proposed approach is shown to significantly outperform other networks in terms of various accuracy measures and processing speed.