MDCC-Net: Multiscale double-channel convolution U-Net framework for colorectal tumor segmentation

A Colorectal Coordinate-Driven Method for Colorectum and Colorectal Cancer Segmentation in Conventional CT Scans

Automated colorectal cancer (CRC) segmentation in medical imaging is the key to achieving automation of CRC detection, staging, and treatment response monitoring. Compared with magnetic resonance imaging (MRI) and computed tomography colonography (CTC), conventional computed tomography (CT) has enormous potential because of its broad implementation, superiority for the hollow viscera (colon), and convenience without needing bowel preparation. However, the segmentation of CRC in conventional CT is more challenging due to the difficulties presenting with the unprepared bowel, such as distinguishing the colorectum from other structures with similar appearance and distinguishing the CRC from the contents of the colorectum. To tackle these challenges, we introduce DeepCRC-SL, the first automated segmentation algorithm for CRC and colorectum in conventional contrast-enhanced CT scans. We propose a topology-aware deep learning-based approach, which builds a novel 1-D colorectal coordinate system and encodes each voxel of the colorectum with a relative position along the coordinate system. We then induce an auxiliary regression task to predict the colorectal coordinate value of each voxel, aiming to integrate global topology into the segmentation network and thus improve the colorectum's continuity. Self-attention layers are utilized to capture global contexts for the coordinate regression task and enhance the ability to differentiate CRC and colorectum tissues. Moreover, a coordinate-driven self-learning (SL) strategy is introduced to leverage a large amount of unlabeled data to improve segmentation performance. We validate the proposed approach on a dataset including 227 labeled and 585 unlabeled CRC cases by fivefold cross-validation. Experimental results demonstrate that our method outperforms some recent related segmentation methods and achieves the segmentation accuracy in DSC for CRC of 0.669 and colorectum of 0.892, reaching to the performance (at 0.639 and 0.890, respectively) of a medical resident with two years of specialized CRC imaging fellowship.

Preoperative clinical radiomics model based on deep learning in prognostic assessment of patients with gallbladder carcinoma

OBJECTIVE

We aimed to develop a preoperative clinical radiomics survival prediction model based on the radiomics features via deep learning to provide a reference basis for preoperative assessment and treatment decisions for patients with gallbladder carcinoma (GBC).

METHODS

A total of 168 GBC patients who underwent preoperative upper abdominal enhanced CT from one high-volume medical center between January 2011 to December 2020 were retrospectively analyzed. The region of interest (ROI) was manually outlined by two physicians using 3D Slicer software to establish a nnU-Net model. The DeepSurv survival prediction model was developed by combining radiomics features and preoperative clinical variables.

RESULTS

A total of 1502 radiomics features were extracted from the ROI results based on the nnU-Net model and manual segmentation, and 13 radiomics features were obtained through the 4-step dimensionality reduction methods, respectively. The C-index and AUC of 1-, 2-, and 3-year survival prediction for the nnU-Net based clinical radiomics DeepSurv model was higher than clinical and nnU-Net based radiomics DeepSurv models in the training and testing sets, and close to manual based clinical radiomics DeepSurv model. Delong-test was performed on the AUC of 1-, 2-, and 3-year survival prediction for the two preoperative clinical radiomics DeepSurv prediction models in the testing set, and the results showed that the two models had the same prediction efficiency (all P > 0.05).

CONCLUSIONS

By using the DeepSurv model via nnU-Net segmentation, postoperative survival outcomes for individual gallbladder carcinoma patients could be assessed and stratified, which can provide references for preoperative diagnosis and treatment decisions.

Deep learning for colorectal cancer detection in contrast-enhanced CT without bowel preparation: a retrospective, multicentre study

BACKGROUND

Contrast-enhanced CT scans provide a means to detect unsuspected colorectal cancer. However, colorectal cancers in contrast-enhanced CT without bowel preparation may elude detection by radiologists. We aimed to develop a deep learning (DL) model for accurate detection of colorectal cancer, and evaluate whether it could improve the detection performance of radiologists.

METHODS

We developed a DL model using a manually annotated dataset (1196 cancer vs 1034 normal). The DL model was tested using an internal test set (98 vs 115), two external test sets (202 vs 265 in 1, and 252 vs 481 in 2), and a real-world test set (53 vs 1524). We compared the detection performance of the DL model with radiologists, and evaluated its capacity to enhance radiologists' detection performance.

FINDINGS

In the four test sets, the DL model had the area under the receiver operating characteristic curves (AUCs) ranging between 0.957 and 0.994. In both the internal test set and external test set 1, the DL model yielded higher accuracy than that of radiologists (97.2% vs 86.0%, p < 0.0001; 94.9% vs 85.3%, p < 0.0001), and significantly improved the accuracy of radiologists (93.4% vs 86.0%, p < 0.0001; 93.6% vs 85.3%, p < 0.0001). In the real-world test set, the DL model delivered sensitivity comparable to that of radiologists who had been informed about clinical indications for most cancer cases (94.3% vs 96.2%, p > 0.99), and it detected 2 cases that had been missed by radiologists.

INTERPRETATION

The developed DL model can accurately detect colorectal cancer and improve radiologists' detection performance, showing its potential as an effective computer-aided detection tool.

FUNDING

This study was supported by National Science Fund for Distinguished Young Scholars of China (No. 81925023); Regional Innovation and Development Joint Fund of National Natural Science Foundation of China (No. U22A20345); National Natural Science Foundation of China (No. 82072090 and No. 82371954); Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application (No. 2022B1212010011); High-level Hospital Construction Project (No. DFJHBF202105).

A Comparative Review on Enhancing Visual Simultaneous Localization and Mapping with Deep Semantic Segmentation

Visual simultaneous localization and mapping (VSLAM) enhances the navigation of autonomous agents in unfamiliar environments by progressively constructing maps and estimating poses. However, conventional VSLAM pipelines often exhibited degraded performance in dynamic environments featuring mobile objects. Recent research in deep learning led to notable progress in semantic segmentation, which involves assigning semantic labels to image pixels. The integration of semantic segmentation into VSLAM can effectively differentiate between static and dynamic elements in intricate scenes. This paper provided a comprehensive comparative review on leveraging semantic segmentation to improve major components of VSLAM, including visual odometry, loop closure detection, and environmental mapping. Key principles and methods for both traditional VSLAM and deep semantic segmentation were introduced. This paper presented an overview and comparative analysis of the technical implementations of semantic integration across various modules of the VSLAM pipeline. Furthermore, it examined the features and potential use cases associated with the fusion of VSLAM and semantics. It was found that the existing VSLAM model continued to face challenges related to computational complexity. Promising future research directions were identified, including efficient model design, multimodal fusion, online adaptation, dynamic scene reconstruction, and end-to-end joint optimization. This review shed light on the emerging paradigm of semantic VSLAM and how deep learning-enabled semantic reasoning could unlock new capabilities for autonomous intelligent systems to operate reliably in the real world.

SG-Transunet: A segmentation-guided Transformer U-Net model for KRAS gene mutation status identification in colorectal cancer

Accurately identifying the Kirsten rat sarcoma virus (KRAS) gene mutation status in colorectal cancer (CRC) patients can assist doctors in deciding whether to use specific targeted drugs for treatment. Although deep learning methods are popular, they are often affected by redundant features from non-lesion areas. Moreover, existing methods commonly extract spatial features from imaging data, which neglect important frequency domain features and may degrade the performance of KRAS gene mutation status identification. To address this deficiency, we propose a segmentation-guided Transformer U-Net (SG-Transunet) model for KRAS gene mutation status identification in CRC. Integrating the strength of convolutional neural networks (CNNs) and Transformers, SG-Transunet offers a unique approach for both lesion segmentation and KRAS mutation status identification. Specifically, for precise lesion localization, we employ an encoder-decoder to obtain segmentation results and guide the KRAS gene mutation status identification task. Subsequently, a frequency domain supplement block is designed to capture frequency domain features, integrating it with high-level spatial features extracted in the encoding path to derive advanced spatial-frequency domain features. Furthermore, we introduce a pre-trained Xception block to mitigate the risk of overfitting associated with small-scale datasets. Following this, an aggregate attention module is devised to consolidate spatial-frequency domain features with global information extracted by the Transformer at shallow and deep levels, thereby enhancing feature discriminability. Finally, we propose a mutual-constrained loss function that simultaneously constrains the segmentation mask acquisition and gene status identification process. Experimental results demonstrate the superior performance of SG-Transunet over state-of-the-art methods in discriminating KRAS gene mutation status.

Fully semantic segmentation for rectal cancer based on post-nCRT MRl modality and deep learning framework

PURPOSE

Rectal tumor segmentation on post neoadjuvant chemoradiotherapy (nCRT) magnetic resonance imaging (MRI) has great significance for tumor measurement, radiomics analysis, treatment planning, and operative strategy. In this study, we developed and evaluated segmentation potential exclusively on post-chemoradiation T2-weighted MRI using convolutional neural networks, with the aim of reducing the detection workload for radiologists and clinicians.

METHODS

A total of 372 consecutive patients with LARC were retrospectively enrolled from October 2015 to December 2017. The standard-of-care neoadjuvant process included 22-fraction intensity-modulated radiation therapy and oral capecitabine. Further, 243 patients (3061 slices) were grouped into training and validation datasets with a random 80:20 split, and 41 patients (408 slices) were used as the test dataset. A symmetric eight-layer deep network was developed using the nnU-Net Framework, which outputs the segmentation result with the same size. The trained deep learning (DL) network was examined using fivefold cross-validation and tumor lesions with different TRGs.

RESULTS

At the stage of testing, the Dice similarity coefficient (DSC), 95% Hausdorff distance (HD95), and mean surface distance (MSD) were applied to quantitatively evaluate the performance of generalization. Considering the test dataset (41 patients, 408 slices), the average DSC, HD95, and MSD were 0.700 (95% CI: 0.680-0.720), 17.73 mm (95% CI: 16.08-19.39), and 3.11 mm (95% CI: 2.67-3.56), respectively. Eighty-two percent of the MSD values were less than 5 mm, and fifty-five percent were less than 2 mm (median 1.62 mm, minimum 0.07 mm).

CONCLUSIONS

The experimental results indicated that the constructed pipeline could achieve relatively high accuracy. Future work will focus on assessing the performances with multicentre external validation.

TdDS-UNet: top-down deeply supervised U-Net for the delineation of 3D colorectal cancer

Automatically delineating colorectal cancers with fuzzy boundaries from 3D images is a challenging task, but the problem of fuzzy boundary delineation in existing deep learning-based methods have not been investigated in depth. Here, an encoder-decoder-based U-shaped network (U-Net) based on top-down deep supervision (TdDS) was designed to accurately and automatically delineate the fuzzy boundaries of colorectal cancer. TdDS refines the semantic targets of the upper and lower stages by mapping ground truths that are more consistent with the stage properties than upsampling deep supervision. This stage-specific approach can guide the model to learn a coarse-to-fine delineation process and improve the delineation accuracy of fuzzy boundaries by gradually shrinking the boundaries. Experimental results showed that TdDS is more customizable and plays a role similar to the attentional mechanism, and it can further improve the capability of the model to delineate colorectal cancer contours. A total of 103, 12, and 29 3D pelvic magnetic resonance imaging volumes were used for training, validation, and testing, respectively. The comparative results indicate that the proposed method exhibits the best comprehensive performance, with a dice similarity coefficient (DSC) of 0.805 ± 0.053 and a hausdorff distance (HD) of 9.28 ± 5.14 voxels. In the delineation performance analysis section also showed that 44.49% of the delineation results are satisfactory and do not require revisions. This study can provide new technical support for the delineation of 3D colorectal cancer. Our method is open source, and the code is available athttps://github.com/odindis/TdDS/tree/main.

Low-light image enhancement based on Retinex-Net with color restoration

Low-light images often suffer from a variety of degradation problems such as loss of detail, color distortions, and prominent noise. In this paper, the Retinex-Net model and loss function with color restoration are proposed to reduce color distortion in low-light image enhancement. The model trains the decom-net and color recovery-net to achieve decomposition of low-light images and color restoration of reflected images, respectively. First, a convolutional neural network and the designed loss functions are used in the decom-net to decompose the low-light image pair into an optimal reflection image and illumination image as the input of the network, and the reflection image after normal light decomposition is taken as the label. Then, an end-to-end color recovery network with a simplified model and time complexity is learned and combined with the color recovery loss function to obtain the correction reflection map with higher perception quality, and gamma correction is applied to the decomposed illumination image. Finally, the corrected reflection image and the illumination image are synthesized to get the enhanced image. The experimental results show that the proposed network model has lower brightness-order-error (LOE) and natural image quality evaluator (NIQE) values, and the average LOE and NIQE values of the low-light dataset images can be reduced to 942 and 6.42, respectively, which significantly improves image quality compared with other low-light enhancement methods. Generally, our proposed method can effectively improve image illuminance and restore color information in the end-to-end learning process of low-light images.

Pixel-Level Classification of Five Histologic Patterns of Lung Adenocarcinoma

Lung adenocarcinoma is the most common histologic type of lung cancer. The pixel-level labeling of histologic patterns of lung adenocarcinoma can assist pathologists in determining tumor grading with more details than normal classification. We manually annotated a dataset containing a total of 1000 patches (200 patches for each pattern) of 512 × 512 pixels and 420 patches (contains test sets) of 1024 × 1024 pixels according to the morphological features of the five histologic patterns of lung adenocarcinoma (lepidic, acinar, papillary, micropapillary, and solid). To generate an even large amount of data patches, we developed a data stitching strategy as a data augmentation for classification in model training. Stitched patches improve the Dice similarity coefficient (DSC) scores by 24.06% on the whole-slide image (WSI) with the solid pattern. We propose a WSI analysis framework for lung adenocarcinoma pathology, intelligently labeling lung adenocarcinoma histologic patterns at the pixel level. Our framework contains five branches of deep neural networks for segmenting each histologic pattern. We test our framework with 200 unclassified patches. The DSC scores of our results outpace comparing networks (U-Net, LinkNet, and FPN) by up to 10.78%. We also perform results on four WSIs with an overall accuracy of 99.6%, demonstrating that our network framework exhibits better accuracy and robustness in most cases.

Deep learning on lateral flow immunoassay for the analysis of detection data

Lateral flow immunoassay (LFIA) is an important detection method in vitro diagnosis, which has been widely used in medical industry. It is difficult to analyze all peak shapes through classical methods due to the complexity of LFIA. Classical methods are generally some peak-finding methods, which cannot distinguish the difference between normal peak and interference or noise peak, and it is also difficult for them to find the weak peak. Here, a novel method based on deep learning was proposed, which can effectively solve these problems. The method had two steps. The first was to classify the data by a classification model and screen out double-peaks data, and second was to realize segmentation of the integral regions through an improved U-Net segmentation model. After training, the accuracy of the classification model for validation set was 99.59%, and using combined loss function (WBCE + DSC), intersection over union (IoU) value of segmentation model for validation set was 0.9680. This method was used in a hand-held fluorescence immunochromatography analyzer designed independently by our team. A Ferritin standard curve was created, and the T/C value correlated well with standard concentrations in the range of 0-500 ng/ml (R ² = 0.9986). The coefficients of variation (CVs) were ≤ 1.37%. The recovery rate ranged from 96.37 to 105.07%. Interference or noise peaks are the biggest obstacle in the use of hand-held instruments, and often lead to peak-finding errors. Due to the changeable and flexible use environment of hand-held devices, it is not convenient to provide any technical support. This method greatly reduced the failure rate of peak finding, which can reduce the customer's need for instrument technical support. This study provided a new direction for the data-processing of point-of-care testing (POCT) instruments based on LFIA.

DMFF-Net: A dual encoding multiscale feature fusion network for ovarian tumor segmentation

Ovarian cancer is a serious threat to the female reproductive system. Precise segmentation of the tumor area helps the doctors to further diagnose the disease. Automatic segmentation techniques for abstracting high-quality features from images through autonomous learning of model have become a hot research topic nowadays. However, the existing methods still have the problem of poor segmentation of ovarian tumor details. To cope with this problem, a dual encoding based multiscale feature fusion network (DMFF-Net) is proposed for ovarian tumor segmentation. Firstly, a dual encoding method is proposed to extract diverse features. These two encoding paths are composed of residual blocks and single dense aggregation blocks, respectively. Secondly, a multiscale feature fusion block is proposed to generate more advanced features. This block constructs feature fusion between two encoding paths to alleviate the feature loss during deep extraction and further increase the information content of the features. Finally, coordinate attention is added to the decoding stage after the feature concatenation, which enables the decoding stage to capture the valid information accurately. The test results show that the proposed method outperforms existing medical image segmentation algorithms for segmenting lesion details. Moreover, the proposed method also performs well in two other segmentation tasks.

Predicting colorectal cancer tumor mutational burden from histopathological images and clinical information using multi-modal deep learning

MOTIVATION

Tumor mutational burden (TMB) is an indicator of the efficacy and prognosis of immune checkpoint therapy in colorectal cancer (CRC). In general, patients with higher TMB values are more likely to benefit from immunotherapy. Though whole-exome sequencing is considered the gold standard for determining TMB, it is difficult to be applied in clinical practice due to its high cost. There are also a few DNA panel-based methods to estimate TMB; however, their detection cost is also high, and the associated wet-lab experiments usually take days, which emphasize the need for faster and cheaper alternatives.

RESULTS

In this study, we propose a multi-modal deep learning model based on a residual network (ResNet) and multi-modal compact bilinear pooling to predict TMB status (i.e. TMB high (TMB_H) or TMB low(TMB_L)) directly from histopathological images and clinical data. We applied the model to CRC data from The Cancer Genome Atlas and compared it with four other popular methods, namely, ResNet18, ResNet50, VGG19 and AlexNet. We tested different TMB thresholds, namely, percentiles of 10%, 14.3%, 15%, 16.3%, 20%, 30% and 50%, to differentiate TMB_H and TMB_L.For the percentile of 14.3% (i.e. TMB value 20) and ResNet18, our model achieved an area under the receiver operating characteristic curve of 0.817 after 5-fold cross-validation, which was better than that of other compared models. In addition, we also found that TMB values were significantly associated with the tumor stage and N and M stages. Our study shows that deep learning models can predict TMB status from histopathological images and clinical information only, which is worth clinical application.

Semantic enhanced Top-k similarity search on weighted HIN

Similarity searches on heterogeneous information networks (HINs) have attracted wide attention from both industrial and academic areas in recent years; for example, they have been used for friend detection in social networks and collaborator recommendation in coauthor networks. The structural information on the HIN can be captured by multiple metapaths, and people usually utilize metapaths to design methods for similarity search. The rich semantics in HINs are not only structural information but also content stored in nodes. However, the content similarity of nodes was usually not valued in the existing methods. Although some researchers have recently considered both types of information in machine learning-based methods for similarity search, they have used structure and content information separately. To address this issue by balancing the influence of structure and content information flexibly in the process of searching, we propose a double channel convolutional neural network model for top-k similarity search, which uses path instances as model inputs and generates structure and content embeddings for nodes based on different metapaths. We design an attention mechanism to enhance the differences in metapaths for each node. Another attention mechanism is used to combine the content and structure information of nodes. Finally, an importance evaluation function is designed to improve the accuracy and make the model more explainable. The experimental results show that our search algorithm can effectively support top-k similarity search in HINs and achieve higher performance than existing approaches.

Attention based multi-scale parallel network for polyp segmentation

Colonoscopy is an effective method for detecting colorectal polyps and preventing colorectal cancer. Therefore, in clinical practice, it is very important to accurately segment the location and shape of polyps from colorectal images, which can effectively assist clinicians in their diagnosis. However, the varying sizes and shapes of colorectal polyps and the fact that the polyps to be segmented are very small and closely resemble their surroundings make this a challenging task. To address these challenges, we propose a parallel network for multi-scale attention decoding-AMNet. We first perform multi-scale fusion of the high-level feature information extracted from the backbone network using upsampling and downsampling, while aggregating the high-level feature information to generate an initial predictive segmentation map for subsequent contextual guidance. Using a parallel attention module as well as a reverse fusion module, relationships between regions and boundaries are established to further refine the edge information and improve the accuracy of the segmentation. Through extensive experiments on four publicly available polyp segmentation datasets, it has been demonstrated that our AMNet is effective in improving the accuracy of polyp segmentation.

U-Net-Based Medical Image Segmentation

Deep learning has been extensively applied to segmentation in medical imaging. U-Net proposed in 2015 shows the advantages of accurate segmentation of small targets and its scalable network architecture. With the increasing requirements for the performance of segmentation in medical imaging in recent years, U-Net has been cited academically more than 2500 times. Many scholars have been constantly developing the U-Net architecture. This paper summarizes the medical image segmentation technologies based on the U-Net structure variants concerning their structure, innovation, efficiency, etc.; reviews and categorizes the related methodology; and introduces the loss functions, evaluation parameters, and modules commonly applied to segmentation in medical imaging, which will provide a good reference for the future research.

Preliminary study of generalized semiautomatic segmentation for 3D voxel labeling of lesions based on deep learning

PURPOSE

The three-dimensional (3D) voxel labeling of lesions requires significant radiologists' effort in the development of computer-aided detection software. To reduce the time required for the 3D voxel labeling, we aimed to develop a generalized semiautomatic segmentation method based on deep learning via a data augmentation-based domain generalization framework. In this study, we investigated whether a generalized semiautomatic segmentation model trained using two types of lesion can segment previously unseen types of lesion.

METHODS

We targeted lung nodules in chest CT images, liver lesions in hepatobiliary-phase images of Gd-EOB-DTPA-enhanced MR imaging, and brain metastases in contrast-enhanced MR images. For each lesion, the 32 × 32 × 32 isotropic volume of interest (VOI) around the center of gravity of the lesion was extracted. The VOI was input into a 3D U-Net model to define the label of the lesion. For each type of target lesion, we compared five types of data augmentation and two types of input data.

RESULTS

For all considered target lesions, the highest dice coefficients among the training patterns were obtained when using a combination of the existing data augmentation-based domain generalization framework and random monochrome inversion and when using the resized VOI as the input image. The dice coefficients were 0.639 ± 0.124 for the lung nodules, 0.660 ± 0.137 for the liver lesions, and 0.727 ± 0.115 for the brain metastases.

CONCLUSIONS

Our generalized semiautomatic segmentation model could label unseen three types of lesion with different contrasts from the surroundings. In addition, the resized VOI as the input image enables the adaptation to the various sizes of lesions even when the size distribution differed between the training set and the test set.