2D to 3D Evolutionary Deep Convolutional Neural Networks for Medical Image Segmentation

Deep Learning Model for Grading and Localization of Lumbar Disc Herniation on Magnetic Resonance Imaging

BACKGROUND

Methods for grading and localization of lumbar disc herniation (LDH) on MRI are complex, time-consuming, and subjective. Utilizing deep learning (DL) models as assistance would mitigate such complexities.

PURPOSE

To develop an interpretable DL model capable of grading and localizing LDH.

STUDY TYPE

Retrospective.

SUBJECTS

1496 patients (M/F: 783/713) were evaluated, and randomly divided into training (70%), validation (10%), and test (20%) sets.

FIELD STRENGTH/SEQUENCE

1.5T MRI for axial T2-weighted sequences (spin echo).

ASSESSMENT

The training set was annotated by three spinal surgeons using the Michigan State University classification to train the DL model. The test set was annotated by a spinal surgery expert (as ground truth labels), and two spinal surgeons (comparison with the trained model). An external test set was employed to evaluate the generalizability of the DL model.

STATISTICAL TESTS

Calculated intersection over union (IoU) for detection consistency, utilized Gwet's AC1 to assess interobserver agreement, and evaluated model performance based on sensitivity and specificity, with statistical significance set at P < 0.05.

RESULTS

The DL model achieved high detection consistency in both the internal test dataset (grading: mean IoU 0.84, recall 99.6%; localization: IoU 0.82, recall 99.5%) and external test dataset (grading: 0.72, 98.0%; localization: 0.71, 97.6%). For internal testing, the DL model (grading: 0.81; localization: 0.76), Rater 1 (0.88; 0.82), and Rater 2 (0.86; 0.83) demonstrated results highly consistent with the ground truth labels. The overall sensitivity of the DL model was 87.0% for grading and 84.0% for localization, while the specificity was 95.5% and 94.4%. For external testing, the DL model showed an appreciable decrease in consistency (grading: 0.69; localization: 0.66), sensitivity (77.2%; 76.7%), and specificity (92.3%; 91.8%).

DATA CONCLUSION

The classification capabilities of the DL model closely resemble those of spinal surgeons. For future improvement, enriching the diversity of cases could enhance the model's generalization.

LEVEL OF EVIDENCE: 4

TECHNICAL EFFICACY

Stage 2.

A review of AutoML optimization techniques for medical image applications

Automatic analysis of medical images using machine learning techniques has gained significant importance over the years. A large number of approaches have been proposed for solving different medical image analysis tasks using machine learning and deep learning approaches. These approaches are quite effective thanks to their ability to analyze large volume of medical imaging data. Moreover, they can also identify patterns that may be difficult for human experts to detect. Manually designing and tuning the parameters of these algorithms is a challenging and time-consuming task. Furthermore, designing a generalized model that can handle different imaging modalities is difficult, as each modality has specific characteristics. To solve these problems and automate the whole pipeline of different medical image analysis tasks, numerous Automatic Machine Learning (AutoML) techniques have been proposed. These techniques include Hyper-parameter Optimization (HPO), Neural Architecture Search (NAS), and Automatic Data Augmentation (ADA). This study provides an overview of several AutoML-based approaches for different medical imaging tasks in terms of optimization search strategies. The usage of optimization techniques (evolutionary, gradient-based, Bayesian optimization, etc.) is of significant importance for these AutoML approaches. We comprehensively reviewed existing AutoML approaches, categorized them, and performed a detailed analysis of different proposed approaches. Furthermore, current challenges and possible future research directions are also discussed.

Deep-learning-based method for the segmentation of ureter and renal pelvis on non-enhanced CT scans

This study aimed to develop a deep-learning (DL) based method for three-dimensional (3D) segmentation of the upper urinary tract (UUT), including ureter and renal pelvis, on non-enhanced computed tomography (NECT) scans. A total of 150 NECT scans with normal appearance of the left UUT were chosen for this study. The dataset was divided into training (n = 130) and validation sets (n = 20). The test set contained 29 randomly chosen cases with computed tomography urography (CTU) and NECT scans, all with normal appearance of the left UUT. An experienced radiologist marked out the left renal pelvis and ureter on each scan. Two types of frameworks (entire and sectional) with three types of DL models (basic UNet, UNet3 + and ViT-UNet) were developed, and evaluated. The sectional framework with basic UNet model achieved the highest mean precision (85.5%) and mean recall (71.9%) on the test set compared to all other tested methods. Compared with CTU scans, this method had higher axial UUT recall than CTU (82.5% vs 69.1%, P < 0.01). This method achieved similar or better visualization of UUT than CTU in many cases, however, in some cases, it exhibited a non-ignorable false-positive rate. The proposed DL method demonstrates promising potential in automated 3D UUT segmentation on NECT scans. The proposed DL models could remarkably improve the efficiency of UUT reconstruction, and have the potential to save many patients from invasive examinations such as CTU. DL models could also serve as a valuable complement to CTU.

SASAN: Spectrum-Axial Spatial Approach Networks for Medical Image Segmentation

Ophthalmic diseases such as central serous chorioretinopathy (CSC) significantly impair the vision of millions of people globally. Precise segmentation of choroid and macular edema is critical for diagnosing and treating these conditions. However, existing 3D medical image segmentation methods often fall short due to the heterogeneous nature and blurry features of these conditions, compounded by medical image clarity issues and noise interference arising from equipment and environmental limitations. To address these challenges, we propose the Spectrum Analysis Synergy Axial-Spatial Network (SASAN), an approach that innovatively integrates spectrum features using the Fast Fourier Transform (FFT). SASAN incorporates two key modules: the Frequency Integrated Neural Enhancer (FINE), which mitigates noise interference, and the Axial-Spatial Elementum Multiplier (ASEM), which enhances feature extraction. Additionally, we introduce the Self-Adaptive Multi-Aspect Loss ( LSM ), which balances image regions, distribution, and boundaries, adaptively updating weights during training. We compiled and meticulously annotated the Choroid and Macular Edema OCT Mega Dataset (CMED-18k), currently the world's largest dataset of its kind. Comparative analysis against 13 baselines shows our method surpasses these benchmarks, achieving the highest Dice scores and lowest HD95 in the CMED and OIMHS datasets. Our code is publicly available at https://github.com/IMOP-lab/SASAN-Pytorch.

Medical image identification methods: A review

The identification of medical images is an essential task in computer-aided diagnosis, medical image retrieval and mining. Medical image data mainly include electronic health record data and gene information data, etc. Although intelligent imaging provided a good scheme for medical image analysis over traditional methods that rely on the handcrafted features, it remains challenging due to the diversity of imaging modalities and clinical pathologies. Many medical image identification methods provide a good scheme for medical image analysis. The concepts pertinent of methods, such as the machine learning, deep learning, convolutional neural networks, transfer learning, and other image processing technologies for medical image are analyzed and summarized in this paper. We reviewed these recent studies to provide a comprehensive overview of applying these methods in various medical image analysis tasks, such as object detection, image classification, image registration, segmentation, and other tasks. Especially, we emphasized the latest progress and contributions of different methods in medical image analysis, which are summarized base on different application scenarios, including classification, segmentation, detection, and image registration. In addition, the applications of different methods are summarized in different application area, such as pulmonary, brain, digital pathology, brain, skin, lung, renal, breast, neuromyelitis, vertebrae, and musculoskeletal, etc. Critical discussion of open challenges and directions for future research are finally summarized. Especially, excellent algorithms in computer vision, natural language processing, and unmanned driving will be applied to medical image recognition in the future.

Artificial Intelligence Image Recognition System for Preventing Wrong-Site Upper Limb Surgery

Our image recognition system employs a deep learning model to differentiate between the left and right upper limbs in images, allowing doctors to determine the correct surgical position. From the experimental results, it was found that the precision rate and the recall rate of the intelligent image recognition system for preventing wrong-site upper limb surgery proposed in this paper could reach 98% and 93%, respectively. The results proved that our Artificial Intelligence Image Recognition System (AIIRS) could indeed assist orthopedic surgeons in preventing the occurrence of wrong-site left and right upper limb surgery. At the same time, in future, we will apply for an IRB based on our prototype experimental results and we will conduct the second phase of human trials. The results of this research paper are of great benefit and research value to upper limb orthopedic surgery.

EU-Net: Automatic U-Net neural architecture search with differential evolutionary algorithm for medical image segmentation

Medical images are crucial in clinical practice, providing essential information for patient assessment and treatment planning. However, manual extraction of information from images is both time-consuming and prone to errors. The emergence of U-Net addresses this challenge by automating the segmentation of anatomical structures and pathological lesions in medical images, thereby significantly enhancing the accuracy of image interpretation and diagnosis. However, the performance of U-Net largely depends on its encoder-decoder structure, which requires researchers with knowledge of neural network architecture design and an in-depth understanding of medical images. In this paper, we propose an automatic U-Net Neural Architecture Search (NAS) algorithm using the differential evolutionary (DE) algorithm, named EU-Net, to segment critical information in medical images to assist physicians in diagnosis. Specifically, by presenting the variable-length strategy, the proposed EU-Net algorithm can sufficiently and automatically search for the neural network architecture without expertise. Moreover, the utilization of crossover, mutation, and selection strategies of DE takes account of the trade-off between exploration and exploitation in the search space. Finally, in the encoding and decoding phases of the proposed algorithm, different block-based and layer-based structures are introduced for architectural optimization. The proposed EU-Net algorithm is validated on two widely used medical datasets, i.e., CHAOS and BUSI, for image segmentation tasks. Extensive experimental results show that the proposed EU-Net algorithm outperforms the chosen peer competitors in both two datasets. In particular, compared to the original U-Net, our proposed method improves the metric mIou by at least 6%.

Automatic segmentation of white matter hyperintensities and correlation analysis for cerebral small vessel disease

Objective

Cerebral white matter hyperintensity can lead to cerebral small vessel disease, MRI images in the brain are used to assess the degree of pathological changes in white matter regions. In this paper, we propose a framework for automatic 3D segmentation of brain white matter hyperintensity based on MRI images to address the problems of low accuracy and segmentation inhomogeneity in 3D segmentation. We explored correlation analyses of cognitive assessment parameters and multiple comparison analyses to investigate differences in brain white matter hyperintensity volume among three cognitive states, Dementia, MCI and NCI. The study explored the correlation between cognitive assessment coefficients and brain white matter hyperintensity volume.

Methods

This paper proposes an automatic 3D segmentation framework for white matter hyperintensity using a deep multi-mapping encoder-decoder structure. The method introduces a 3D residual mapping structure for the encoder and decoder. Multi-layer Cross-connected Residual Mapping Module (MCRCM) is proposed in the encoding stage to enhance the expressiveness of model and perception of detailed features. Spatial Attention Weighted Enhanced Supervision Module (SAWESM) is proposed in the decoding stage to adjust the supervision strategy through a spatial attention weighting mechanism. This helps guide the decoder to perform feature reconstruction and detail recovery more effectively.

Result

Experimental data was obtained from a privately owned independent brain white matter dataset. The results of the automatic 3D segmentation framework showed a higher segmentation accuracy compared to nnunet and nnunet-resnet, with a p-value of <0.001 for the two cognitive assessment parameters MMSE and MoCA. This indicates that larger brain white matter are associated with lower scores of MMSE and MoCA, which in turn indicates poorer cognitive function. The order of volume size of white matter hyperintensity in the three groups of cognitive states is dementia, MCI and NCI, respectively.

Conclusion

The paper proposes an automatic 3D segmentation framework for brain white matter that achieves high-precision segmentation. The experimental results show that larger volumes of segmented regions have a negative correlation with lower scoring coefficients of MMSE and MoCA. This correlation analysis provides promising treatment prospects for the treatment of cerebral small vessel diseases in the brain through 3D segmentation analysis of brain white matter. The differences in the volume of white matter hyperintensity regions in subjects with three different cognitive states can help to better understand the mechanism of cognitive decline in clinical research.

Deep learning in CT image segmentation of cervical cancer: a systematic review and meta-analysis

Background

This paper attempts to conduct a systematic review and meta-analysis of deep learning (DLs) models for cervical cancer CT image segmentation.

Methods

Relevant studies were systematically searched in PubMed, Embase, The Cochrane Library, and Web of science. The literature on DLs for cervical cancer CT image segmentation were included, a meta-analysis was performed on the dice similarity coefficient (DSC) of the segmentation results of the included DLs models. We also did subgroup analyses according to the size of the sample, type of segmentation (i.e., two dimensions and three dimensions), and three organs at risk (i.e., bladder, rectum, and femur). This study was registered in PROSPERO prior to initiation (CRD42022307071).

Results

A total of 1893 articles were retrieved and 14 articles were included in the meta-analysis. The pooled effect of DSC score of clinical target volume (CTV), bladder, rectum, femoral head were 0.86(95%CI 0.84 to 0.87), 0.91(95%CI 0.89 to 0.93), 0.83(95%CI 0.79 to 0.88), and 0.92(95%CI 0.91to 0.94), respectively. For the performance of segmented CTV by two dimensions (2D) and three dimensions (3D) model, the DSC score value for 2D model was 0.87 (95%CI 0.85 to 0.90), while the DSC score for 3D model was 0.85 (95%CI 0.82 to 0.87). As for the effect of the capacity of sample on segmentation performance, no matter whether the sample size is divided into two groups: greater than 100 and less than 100, or greater than 150 and less than 150, the results show no difference (P > 0.05). Four papers reported the time for segmentation from 15 s to 2 min.

Conclusion

DLs have good accuracy in automatic segmentation of CT images of cervical cancer with a less time consuming and have good prospects for future radiotherapy applications, but still need public high-quality databases and large-scale research verification.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13014-022-02148-6.

Automated segmentation of vertebral cortex with 3D U-Net-based deep convolutional neural network

Objectives: We developed a 3D U-Net-based deep convolutional neural network for the automatic segmentation of the vertebral cortex. The purpose of this study was to evaluate the accuracy of the 3D U-Net deep learning model.

Methods: In this study, a fully automated vertebral cortical segmentation method with 3D U-Net was developed, and ten-fold cross-validation was employed. Through data augmentation, we obtained 1,672 3D images of chest CT scans. Segmentation was performed using a conventional image processing method and manually corrected by a senior radiologist to create the gold standard. To compare the segmentation performance, 3D U-Net, Res U-Net, Ki U-Net, and Seg Net were used to segment the vertebral cortex in CT images. The segmentation performance of 3D U-Net and the other three deep learning algorithms was evaluated using DSC, mIoU, MPA, and FPS.

Results: The DSC, mIoU, and MPA of 3D U-Net are better than the other three strategies, reaching 0.71 ± 0.03, 0.74 ± 0.08, and 0.83 ± 0.02, respectively, indicating promising automated segmentation results. The FPS is slightly lower than that of Seg Net (23.09 ± 1.26 vs. 30.42 ± 3.57).

Conclusion: Cortical bone can be effectively segmented based on 3D U-net.

Hybrid Methodology Based on Symmetrized Dot Pattern and Convolutional Neural Networks for Fault Diagnosis of Power Cables

This study proposes a recognition method based on symmetrized dot pattern (SDP) analysis and convolutional neural network (CNN) for rapid and accurate diagnosis of insulation defect problems by detecting the partial discharge (PD) signals of XLPE power cables. First, a normal and three power cable models with different insulation defects are built. The PD signals resulting from power cable insulation defects are measured. The frequency and amplitude variations of PD signals from different defects are reflected by comprehensible images using the proposed SDP analysis method. The features of different power cable defects are presented. Finally, the feature image is trained and identified by CNN to achieve a power cable insulation fault diagnosis system. The experimental results show that the proposed method could accurately diagnose the fault types of power cable insulation defects with a recognition accuracy of 98%. The proposed method is characterized by a short detection time and high diagnostic accuracy. It can effectively detect the power cable PD to identify the fault type of the insulation defect.

Artificial Intelligence-Driven Prediction Modeling and Decision Making in Spine Surgery Using Hybrid Machine Learning Models

Healthcare systems worldwide generate vast amounts of data from many different sources. Although of high complexity for a human being, it is essential to determine the patterns and minor variations in the genomic, radiological, laboratory, or clinical data that reliably differentiate phenotypes or allow high predictive accuracy in health-related tasks. Convolutional neural networks (CNN) are increasingly applied to image data for various tasks. Its use for non-imaging data becomes feasible through different modern machine learning techniques, converting non-imaging data into images before inputting them into the CNN model. Considering also that healthcare providers do not solely use one data modality for their decisions, this approach opens the door for multi-input/mixed data models which use a combination of patient information, such as genomic, radiological, and clinical data, to train a hybrid deep learning model. Thus, this reflects the main characteristic of artificial intelligence: simulating natural human behavior. The present review focuses on key advances in machine and deep learning, allowing for multi-perspective pattern recognition across the entire information set of patients in spine surgery. This is the first review of artificial intelligence focusing on hybrid models for deep learning applications in spine surgery, to the best of our knowledge. This is especially interesting as future tools are unlikely to use solely one data modality. The techniques discussed could become important in establishing a new approach to decision-making in spine surgery based on three fundamental pillars: (1) patient-specific, (2) artificial intelligence-driven, (3) integrating multimodal data. The findings reveal promising research that already took place to develop multi-input mixed-data hybrid decision-supporting models. Their implementation in spine surgery may hence be only a matter of time.

Evolutionary Deep Attention Convolutional Neural Networks for 2D and 3D Medical Image Segmentation

Developing a convolutional neural network (CNN) for medical image segmentation is a complex task, especially when dealing with the limited number of available labelled medical images and computational resources. This task can be even more difficult if the aim is to develop a deep network and using a complicated structure like attention blocks. Because of various types of noises, artefacts and diversity in medical images, using complicated network structures like attention mechanism to improve the accuracy of segmentation is inevitable. Therefore, it is necessary to develop techniques to address the above difficulties. Neuroevolution is the combination of evolutionary computation and neural networks to establish a network automatically. However, Neuroevolution is computationally expensive, specifically to create 3D networks. In this paper, an automatic, efficient, accurate, and robust technique is introduced to develop deep attention convolutional neural networks utilising Neuroevolution for both 2D and 3D medical image segmentation. The proposed evolutionary technique can find a very good combination of six attention modules to recover spatial information from downsampling section and transfer them to the upsampling section of a U-Net-based network-six different CT and MRI datasets are employed to evaluate the proposed model for both 2D and 3D image segmentation. The obtained results are compared to state-of-the-art manual and automatic models, while our proposed model outperformed all of them.

CST: A Multitask Learning Framework for Colorectal Cancer Region Mining Based on Transformer

Colorectal cancer is a high death rate cancer until now; from the clinical view, the diagnosis of the tumour region is critical for the doctors. But with data accumulation, this task takes lots of time and labor with large variances between different doctors. With the development of computer vision, detection and segmentation of the colorectal cancer region from CT or MRI image series are a great challenge in the past decades, and there still have great demands on automatic diagnosis. In this paper, we proposed a novel transfer learning protocol, called CST, that is, a union framework for colorectal cancer region detection and segmentation task based on the transformer model, which effectively constructs the cancer region detection and its segmentation jointly. To make a higher detection accuracy, we incorporate an autoencoder-based image-level decision approach that leverages the image-level decision of a cancer slice. We also compared our framework with one-stage and two-stage object detection methods; the results show that our proposed method achieves better results on detection and segmentation tasks. And this proposed framework will give another pathway for colorectal cancer screen by way of artificial intelligence.

SGPNet: A Three-Dimensional Multitask Residual Framework for Segmentation and IDH Genotype Prediction of Gliomas

Glioma is the main type of malignant brain tumor in adults, and the status of isocitrate dehydrogenase (IDH) mutation highly affects the diagnosis, treatment, and prognosis of gliomas. Radiographic medical imaging provides a noninvasive platform for sampling both inter and intralesion heterogeneity of gliomas, and previous research has shown that the IDH genotype can be predicted from the fusion of multimodality radiology images. The features of medical images and IDH genotype are vital for medical treatment; however, it still lacks a multitask framework for the segmentation of the lesion areas of gliomas and the prediction of IDH genotype. In this paper, we propose a novel three-dimensional (3D) multitask deep learning model for segmentation and genotype prediction (SGPNet). The residual units are also introduced into the SGPNet that allows the output blocks to extract hierarchical features for different tasks and facilitate the information propagation. Our model reduces 26.6% classification error rates comparing with previous models on the datasets of Multimodal Brain Tumor Segmentation Challenge (BRATS) 2020 and The Cancer Genome Atlas (TCGA) gliomas’ databases. Furthermore, we first practically investigate the influence of lesion areas on the performance of IDH genotype prediction by setting different groups of learning targets. The experimental results indicate that the information of lesion areas is more important for the IDH genotype prediction. Our framework is effective and generalizable, which can serve as a highly automated tool to be applied in clinical decision making.