Boundary-Weighted Domain Adaptive Neural Network for Prostate MR Image Segmentation

Deep learning based adaptive and automatic measurement of palpebral margin in eyelid morphology

Accurate anatomical measurements of the eyelids are essential in periorbital plastic surgery for both disease treatment and procedural planning. Recent researches in eye diseases have adopted deep learning works to measure MRD. However, such works encounter challenges in practical implementation, and the model accuracy needs to be improved. Here, we have introduced a deep learning-based adaptive and automatic measurement (DeepAAM) by employing the U-Net architecture enhanced through attention mechanisms and multiple algorithms. DeepAAM enables adaptive image recognition and adjustment in practical application, and improves the measurement accuracy of Marginal Reflex Distance (MRD). Meanwhile, it for the first time measures the Margin Iris Intersectant Angle (MIA) as an innovative evaluation index. Besides, this fully automated method surpasses other models in terms of accuracy for iris and sclera segmentation. DeepAAM offers a novel, comprehensive, and objective approach to the quantification of ocular morphology.

Enhancing Physician-Patient Communication in Oncology Using GPT-4 Through Simplified Radiology Reports: Multicenter Quantitative Study

BACKGROUND

Effective physician-patient communication is essential in clinical practice, especially in oncology, where radiology reports play a crucial role. These reports are often filled with technical jargon, making them challenging for patients to understand and affecting their engagement and decision-making. Large language models, such as GPT-4, offer a novel approach to simplifying these reports and potentially enhancing communication and patient outcomes.

OBJECTIVE

We aimed to assess the feasibility and effectiveness of using GPT-4 to simplify oncological radiology reports to improve physician-patient communication.

METHODS

In a retrospective study approved by the ethics review committees of multiple hospitals, 698 radiology reports for malignant tumors produced between October 2023 and December 2023 were analyzed. In total, 70 (10%) reports were selected to develop templates and scoring scales for GPT-4 to create simplified interpretative radiology reports (IRRs). Radiologists checked the consistency between the original radiology reports and the IRRs, while volunteer family members of patients, all of whom had at least a junior high school education and no medical background, assessed readability. Doctors evaluated communication efficiency through simulated consultations.

RESULTS

Transforming original radiology reports into IRRs resulted in clearer reports, with word count increasing from 818.74 to 1025.82 (P<.001), volunteers' reading time decreasing from 674.86 seconds to 589.92 seconds (P<.001), and reading rate increasing from 72.15 words per minute to 104.70 words per minute (P<.001). Physician-patient communication time significantly decreased, from 1116.11 seconds to 745.30 seconds (P<.001), and patient comprehension scores improved from 5.51 to 7.83 (P<.001).

CONCLUSIONS

This study demonstrates the significant potential of large language models, specifically GPT-4, to facilitate medical communication by simplifying oncological radiology reports. Simplified reports enhance patient understanding and the efficiency of doctor-patient interactions, suggesting a valuable application of artificial intelligence in clinical practice to improve patient outcomes and health care communication.

A-Eval: A benchmark for cross-dataset and cross-modality evaluation of abdominal multi-organ segmentation

Although deep learning has revolutionized abdominal multi-organ segmentation, its models often struggle with generalization due to training on small-scale, specific datasets and modalities. The recent emergence of large-scale datasets may mitigate this issue, but some important questions remain unsolved: Can models trained on these large datasets generalize well across different datasets and imaging modalities? If yes/no, how can we further improve their generalizability? To address these questions, we introduce A-Eval, a benchmark for the cross-dataset and cross-modality Evaluation ('Eval') of Abdominal ('A') multi-organ segmentation, integrating seven datasets across CT and MRI modalities. Our evaluations indicate that significant domain gaps persist despite larger data scales. While increased datasets improve generalization, model performance on unseen data remains inconsistent. Joint training across multiple datasets and modalities enhances generalization, though annotation inconsistencies pose challenges. These findings highlight the need for diverse and well-curated training data across various clinical scenarios and modalities to develop robust medical imaging models. The code and pre-trained models are available at https://github.com/uni-medical/A-Eval.

ETDformer: an effective transformer block for segmentation of intracranial hemorrhage

Intracerebral hemorrhage (ICH) medical image segmentation plays a crucial role in clinical diagnostics and treatment planning. The U-Net architecture, known for its encoder-decoder design and skip connections, is widely used but often struggles with accurately delineating complex struct ures like ICH regions. Recently, transformer models have been incorporated into medical image segmentation, improving performance by capturing long-range dependencies. However, existing methods still face challenges in incorrectly segmenting non-target areas and preserving detailed information in the target region. To address these issues, we propose a novel segmentation model that combines U-Net's local feature extraction with the transformer's global perceptiveness. Our method introduces an External Storage Module (ES Module) to capture and store feature similarities between adjacent slices, and a Top-Down Attention (TDAttention) mechanism to focus on relevant lesion regions while enhancing target boundary segmentation. Additionally, we introduce a boundary DoU loss to improve lesion boundary delineation. Evaluations on the intracranial hemorrhage dataset (IHSAH) from the Second Affiliated Hospital of Fujian Medical University, as well as the publicly available Brain Hemorrhage Segmentation Dataset (BHSD), demonstrate that our approach achieves DSC scores of 91.29% and 85.10% on the IHSAH and BHSD datasets, respectively, outperforming the second-best Cascaded MERIT by 2.19% and 2.05%, respectively. Moreover, our method provides enhanced visualization of lesion details, significantly aiding diagnostic accuracy.

Domain adaptation using AdaBN and AdaIN for high-resolution IVD mesh reconstruction from clinical MRI

PURPOSE

Deep learning has firmly established its dominance in medical imaging applications. However, careful consideration must be exercised when transitioning a trained source model to adapt to an entirely distinct environment that deviates significantly from the training set. The majority of the efforts to mitigate this issue have predominantly focused on classification and segmentation tasks. In this work, we perform a domain adaptation of a trained source model to reconstruct high-resolution intervertebral disc meshes from low-resolution MRI.

METHODS

To address the outlined challenges, we use MRI2Mesh as the shape reconstruction network. It incorporates three major modules: image encoder, mesh deformation, and cross-level feature fusion. This feature fusion module is used to encapsulate local and global disc features. We evaluate two major domain adaptation techniques: adaptive batch normalization (AdaBN) and adaptive instance normalization (AdaIN) for the task of shape reconstruction.

RESULTS

Experiments conducted on distinct datasets, including data from different populations, machines, and test sites demonstrate the effectiveness of MRI2Mesh for domain adaptation. MRI2Mesh achieved up to a 14% decrease in Hausdorff distance (HD) and a 19% decrease in the point-to-surface (P2S) metric for both AdaBN and AdaIN experiments, indicating improved performance.

CONCLUSION

MRI2Mesh has demonstrated consistent superiority to the state-of-the-art Voxel2Mesh network across a diverse range of datasets, populations, and scanning protocols, highlighting its versatility. Additionally, AdaBN has emerged as a robust method compared to the AdaIN technique. Further experiments show that MRI2Mesh, when combined with AdaBN, holds immense promise for enhancing the precision of anatomical shape reconstruction in domain adaptation.

Unsupervised domain adaptation based on feature and edge alignment for femur X-ray image segmentation

The gold standard for diagnosing osteoporosis is bone mineral density (BMD) measurement by dual-energy X-ray absorptiometry (DXA). However, various factors during the imaging process cause domain shifts in DXA images, which lead to incorrect bone segmentation. Research shows that poor bone segmentation is one of the prime reasons of inaccurate BMD measurement, severely affecting the diagnosis and treatment plans for osteoporosis. In this paper, we propose a Multi-feature Joint Discriminative Domain Adaptation (MDDA) framework to improve segmentation performance and the generalization of the network in domain-shifted images. The proposed method learns domain-invariant features between the source and target domains from the perspectives of multi-scale features and edges, and is evaluated on real data from multi-center datasets. Compared to other state-of-the-art methods, the feature prior from the source domain and edge prior enable the proposed MDDA to achieve the optimal domain adaptation performance and generalization. It also demonstrates superior performance in domain adaptation tasks on small amount datasets, even using only 5 or 10 images. In this study, MDDA provides an accurate bone segmentation tool for BMD measurement based on DXA imaging.

SGDA: Towards 3-D Universal Pulmonary Nodule Detection via Slice Grouped Domain Attention

Lung cancer is the leading cause of cancer death worldwide. The best solution for lung cancer is to diagnose the pulmonary nodules in the early stage, which is usually accomplished with the aid of thoracic computed tomography (CT). As deep learning thrives, convolutional neural networks (CNNs) have been introduced into pulmonary nodule detection to help doctors in this labor-intensive task and demonstrated to be very effective. However, the current pulmonary nodule detection methods are usually domain-specific, and cannot satisfy the requirement of working in diverse real-world scenarios. To address this issue, we propose a slice grouped domain attention (SGDA) module to enhance the generalization capability of the pulmonary nodule detection networks. This attention module works in the axial, coronal, and sagittal directions. In each direction, we divide the input feature into groups, and for each group, we utilize a universal adapter bank to capture the feature subspaces of the domains spanned by all pulmonary nodule datasets. Then the bank outputs are combined from the perspective of domain to modulate the input group. Extensive experiments demonstrate that SGDA enables substantially better multi-domain pulmonary nodule detection performance compared with the state-of-the-art multi-domain learning methods.

Deep Learning Prostate MRI Segmentation Accuracy and Robustness: A Systematic Review

Purpose To investigate the accuracy and robustness of prostate segmentation using deep learning across various training data sizes, MRI vendors, prostate zones, and testing methods relative to fellowship-trained diagnostic radiologists. Materials and Methods In this systematic review, Embase, PubMed, Scopus, and Web of Science databases were queried for English-language articles using keywords and related terms for prostate MRI segmentation and deep learning algorithms dated to July 31, 2022. A total of 691 articles from the search query were collected and subsequently filtered to 48 on the basis of predefined inclusion and exclusion criteria. Multiple characteristics were extracted from selected studies, such as deep learning algorithm performance, MRI vendor, and training dataset features. The primary outcome was comparison of mean Dice similarity coefficient (DSC) for prostate segmentation for deep learning algorithms versus diagnostic radiologists. Results Forty-eight studies were included. Most published deep learning algorithms for whole prostate gland segmentation (39 of 42 [93%]) had a DSC at or above expert level (DSC ≥ 0.86). The mean DSC was 0.79 ± 0.06 (SD) for peripheral zone, 0.87 ± 0.05 for transition zone, and 0.90 ± 0.04 for whole prostate gland segmentation. For selected studies that used one major MRI vendor, the mean DSCs of each were as follows: General Electric (three of 48 studies), 0.92 ± 0.03; Philips (four of 48 studies), 0.92 ± 0.02; and Siemens (six of 48 studies), 0.91 ± 0.03. Conclusion Deep learning algorithms for prostate MRI segmentation demonstrated accuracy similar to that of expert radiologists despite varying parameters; therefore, future research should shift toward evaluating segmentation robustness and patient outcomes across diverse clinical settings. Keywords: MRI, Genital/Reproductive, Prostate Segmentation, Deep Learning Systematic review registration link: osf.io/nxaev © RSNA, 2024.

Protecting Prostate Cancer Classification From Rectal Artifacts via Targeted Adversarial Training

Magnetic resonance imaging (MRI)-based deep neural networks (DNN) have been widely developed to perform prostate cancer (PCa) classification. However, in real-world clinical situations, prostate MRIs can be easily impacted by rectal artifacts, which have been found to lead to incorrect PCa classification. Existing DNN-based methods typically do not consider the interference of rectal artifacts on PCa classification, and do not design specific strategy to address this problem. In this study, we proposed a novel Targeted adversarial training with Proprietary Adversarial Samples (TPAS) strategy to defend the PCa classification model against the influence of rectal artifacts. Specifically, based on clinical prior knowledge, we generated proprietary adversarial samples with rectal artifact-pattern adversarial noise, which can severely mislead PCa classification models optimized by the ordinary training strategy. We then jointly exploited the generated proprietary adversarial samples and original samples to train the models. To demonstrate the effectiveness of our strategy, we conducted analytical experiments on multiple PCa classification models. Compared with ordinary training strategy, TPAS can effectively improve the single- and multi-parametric PCa classification at patient, slice and lesion level, and bring substantial gains to recent advanced models. In conclusion, TPAS strategy can be identified as a valuable way to mitigate the influence of rectal artifacts on deep learning models for PCa classification.

Catalyzing Precision Medicine: Artificial Intelligence Advancements in Prostate Cancer Diagnosis and Management

BACKGROUND

The aim was to analyze the current state of deep learning (DL)-based prostate cancer (PCa) diagnosis with a focus on magnetic resonance (MR) prostate reconstruction; PCa detection/stratification/reconstruction; positron emission tomography/computed tomography (PET/CT); androgen deprivation therapy (ADT); prostate biopsy; associated challenges and their clinical implications.

METHODS

A search of the PubMed database was conducted based on the inclusion and exclusion criteria for the use of DL methods within the abovementioned areas.

RESULTS

A total of 784 articles were found, of which, 64 were included. Reconstruction of the prostate, the detection and stratification of prostate cancer, the reconstruction of prostate cancer, and diagnosis on PET/CT, ADT, and biopsy were analyzed in 21, 22, 6, 7, 2, and 6 studies, respectively. Among studies describing DL use for MR-based purposes, datasets with magnetic field power of 3 T, 1.5 T, and 3/1.5 T were used in 18/19/5, 0/1/0, and 3/2/1 studies, respectively, of 6/7 studies analyzing DL for PET/CT diagnosis which used data from a single institution. Among the radiotracers, [68Ga]Ga-PSMA-11, [18F]DCFPyl, and [18F]PSMA-1007 were used in 5, 1, and 1 study, respectively. Only two studies that analyzed DL in the context of DT met the inclusion criteria. Both were performed with a single-institution dataset with only manual labeling of training data. Three studies, each analyzing DL for prostate biopsy, were performed with single- and multi-institutional datasets. TeUS, TRUS, and MRI were used as input modalities in two, three, and one study, respectively.

CONCLUSION

DL models in prostate cancer diagnosis show promise but are not yet ready for clinical use due to variability in methods, labels, and evaluation criteria. Conducting additional research while acknowledging all the limitations outlined is crucial for reinforcing the utility and effectiveness of DL-based models in clinical settings.

Few-shot segmentation framework for lung nodules via an optimized active contour model

BACKGROUND

Accurate segmentation of lung nodules is crucial for the early diagnosis and treatment of lung cancer in clinical practice. However, the similarity between lung nodules and surrounding tissues has made their segmentation a longstanding challenge.

PURPOSE

Existing deep learning and active contour models each have their limitations. This paper aims to integrate the strengths of both approaches while mitigating their respective shortcomings.

METHODS

In this paper, we propose a few-shot segmentation framework that combines a deep neural network with an active contour model. We introduce heat kernel convolutions and high-order total variation into the active contour model and solve the challenging nonsmooth optimization problem using the alternating direction method of multipliers. Additionally, we use the presegmentation results obtained from training a deep neural network on a small sample set as the initial contours for our optimized active contour model, addressing the difficulty of manually setting the initial contours.

RESULTS

We compared our proposed method with state-of-the-art methods for segmentation effectiveness using clinical computed tomography (CT) images acquired from two different hospitals and the publicly available LIDC dataset. The results demonstrate that our proposed method achieved outstanding segmentation performance according to both visual and quantitative indicators.

CONCLUSION

Our approach utilizes the output of few-shot network training as prior information, avoiding the need to select the initial contour in the active contour model. Additionally, it provides mathematical interpretability to the deep learning, reducing its dependency on the quantity of training samples.

MicroSegNet: A deep learning approach for prostate segmentation on micro-ultrasound images

Micro-ultrasound (micro-US) is a novel 29-MHz ultrasound technique that provides 3-4 times higher resolution than traditional ultrasound, potentially enabling low-cost, accurate diagnosis of prostate cancer. Accurate prostate segmentation is crucial for prostate volume measurement, cancer diagnosis, prostate biopsy, and treatment planning. However, prostate segmentation on micro-US is challenging due to artifacts and indistinct borders between the prostate, bladder, and urethra in the midline. This paper presents MicroSegNet, a multi-scale annotation-guided transformer UNet model designed specifically to tackle these challenges. During the training process, MicroSegNet focuses more on regions that are hard to segment (hard regions), characterized by discrepancies between expert and non-expert annotations. We achieve this by proposing an annotation-guided binary cross entropy (AG-BCE) loss that assigns a larger weight to prediction errors in hard regions and a lower weight to prediction errors in easy regions. The AG-BCE loss was seamlessly integrated into the training process through the utilization of multi-scale deep supervision, enabling MicroSegNet to capture global contextual dependencies and local information at various scales. We trained our model using micro-US images from 55 patients, followed by evaluation on 20 patients. Our MicroSegNet model achieved a Dice coefficient of 0.939 and a Hausdorff distance of 2.02 mm, outperforming several state-of-the-art segmentation methods, as well as three human annotators with different experience levels. Our code is publicly available at https://github.com/mirthAI/MicroSegNet and our dataset is publicly available at https://zenodo.org/records/10475293.

An auxiliary diagnostic tool for common fundus diseases based on fundus color photography and light-weight classification models

OBJECTIVE

To evaluate the performance of two lightweight neural network models in the diagnosis of common fundus diseases and make comparison to another two classical models.

METHODS

A total of 16,000 color fundus photography were collected, including 2000 each of glaucoma, diabetic retinopathy (DR), high myopia, central retinal vein occlusion (CRVO), age-related macular degeneration (AMD), optic neuropathy, and central serous chorioretinopathy (CSC), in addition to 2000 normal fundus. Fundus photography was obtained from patients or physical examiners who visited the Ophthalmology Department of Beijing Tongren Hospital, Capital Medical University. Each fundus photography has been diagnosed and labeled by two professional ophthalmologists. Two classical classification models (ResNet152 and DenseNet121), and two lightweight classification models (MobileNetV3 and ShufflenetV2), were trained. Area under the curve (AUC), sensitivity, specificity, accuracy, positive predictive value, and negative predictive value were used to evaluate the performance of the four models.

RESULTS

Compared with the classical classification model, the total size and number of parameters of the two lightweight classification models were significantly reduced, and the classification speed was sharply improved. Compared with the DenseNet121 model, the ShufflenetV2 model took 50.7% less time to make a diagnosis on a fundus photography. The classical models performed better than lightweight classification models, and Densenet121 showed highest AUC in five out of the seven common fundus diseases. However, the performance of lightweight classification models is satisfying. The AUCs using MobileNetV3 model to diagnose AMD, diabetic retinopathy, glaucoma, CRVO, high myopia, optic atrophy, and CSC were 0.805, 0.892, 0.866, 0.812, 0.887, 0.868, and 0.803, respectively. For ShufflenetV2model, the AUCs for the above seven diseases were 0.856, 0.893, 0.855, 0.884, 0.891, 0.867, and 0.844, respectively.

CONCLUSION

The training of light-weight neural network models based on color fundus photography for the diagnosis of common fundus diseases is not only fast but also has a significant reduction in storage size and parameter number compared with the classical classification model, and can achieve satisfactory accuracy.

Spatiotemporal knowledge teacher-student reinforcement learning to detect liver tumors without contrast agents

Detecting Liver tumors without contrast agents (CAs) has shown great potential to advance liver cancer screening. It enables the provision of a reliable liver tumor-detecting result from non-enhanced MR images comparable to the radiologists' results from CA-enhanced MR images, thus eliminating the high risk of CAs, preventing an experience gap between radiologists and simplifying clinical workflows. In this paper, we proposed a novel spatiotemporal knowledge teacher-student reinforcement learning (SKT-RL) as a safe, speedy, and inexpensive contrast-free technology for liver tumor detection. Our SKT-RL builds a teacher-student framework to realize the exploring of explicit liver tumor knowledge from a teacher network on clear contrast-enhanced images to guide a student network to detect tumors from non-enhanced images directly. Importantly, our STK-RL enables three novelties in aspects of construction, transferring, and optimization to tumor knowledge to improve the guide effect. (1) A new spatiotemporal ternary knowledge set enables the construction of accurate knowledge that allows understanding of DRL's behavior (what to do) and reason (why to do it) behind reliable detection within each state and between their related historical states. (2) A novel pixel momentum transferring strategy enables detailed and controlled knowledge transfer ability. It transfers knowledge at a pixel level to enlarge the explorable space of transferring and control how much knowledge is transferred to prevent over-rely of the student to the teacher. (3) A phase-trend reward function designs different evaluations according to different detection phases to optimal for each phase in high-precision but also allows reward trend to constraint the evaluation to improve stability. Comprehensive experiments on a generalized liver tumor dataset with 375 patients (including hemangiomas, hepatocellular carcinoma, and normal controls) show that our novel SKT-RL attains a new state-of-the-art performance (improved precision by at least 4% when comparing the six recent advanced methods) in the task of liver tumor detection without CAs. The results proved that our SKT-DRL has greatly promoted the development and deployment of contrast-free liver tumor technology.

Learning generalizable visual representation via adaptive spectral random convolution for medical image segmentation

Medical image segmentation models often fail to generalize well when applied to new datasets, hindering their usage in clinical practice. Existing random-convolution-based domain generalization approaches, which involve randomizing the convolutional kernel weights in the initial layers of CNN models, have shown promise in improving model generalizability. Nevertheless, the indiscriminate introduction of high-frequency noise during early feature extraction may pollute the critical fine details and degrade the model's performance on new datasets. To mitigate this problem, we propose an adaptive spectral random convolution (ASRConv) module designed to selectively randomize low-frequency features while avoiding the introduction of high-frequency artifacts. Unlike prior arts, ASRConv dynamically generates convolution kernel weights, enabling more effective control over feature frequencies than randomized kernels. Specifically, ASRConv achieves this selective randomization through a novel weight generation module conditioned on random noise inputs. The adversarial domain augmentation strategy guides the weight generation module in adaptively suppressing high-frequency noise during training, allowing ASRConv to improve feature diversity and reduce overfitting to specific domains. Extensive experimental results show that our proposed ASRConv method consistently outperforms the state-of-the-art methods, with average DSC improvements of 3.07% and 1.18% on fundus and polyp datasets, respectively. We also qualitatively demonstrate the robustness of our model against domain distribution shifts. All these results validate the effectiveness of the proposed ASRConv in learning domain-invariant representations for robust medical image segmentation.

TISS-net: Brain tumor image synthesis and segmentation using cascaded dual-task networks and error-prediction consistency

Accurate segmentation of brain tumors from medical images is important for diagnosis and treatment planning, and it often requires multi-modal or contrast-enhanced images. However, in practice some modalities of a patient may be absent. Synthesizing the missing modality has a potential for filling this gap and achieving high segmentation performance. Existing methods often treat the synthesis and segmentation tasks separately or consider them jointly but without effective regularization of the complex joint model, leading to limited performance. We propose a novel brain Tumor Image Synthesis and Segmentation network (TISS-Net) that obtains the synthesized target modality and segmentation of brain tumors end-to-end with high performance. First, we propose a dual-task-regularized generator that simultaneously obtains a synthesized target modality and a coarse segmentation, which leverages a tumor-aware synthesis loss with perceptibility regularization to minimize the high-level semantic domain gap between synthesized and real target modalities. Based on the synthesized image and the coarse segmentation, we further propose a dual-task segmentor that predicts a refined segmentation and error in the coarse segmentation simultaneously, where a consistency between these two predictions is introduced for regularization. Our TISS-Net was validated with two applications: synthesizing FLAIR images for whole glioma segmentation, and synthesizing contrast-enhanced T1 images for Vestibular Schwannoma segmentation. Experimental results showed that our TISS-Net largely improved the segmentation accuracy compared with direct segmentation from the available modalities, and it outperformed state-of-the-art image synthesis-based segmentation methods.

The role of AI in prostate MRI quality and interpretation: Opportunities and challenges

Prostate MRI plays an important role in imaging the prostate gland and surrounding tissues, particularly in the diagnosis and management of prostate cancer. With the widespread adoption of multiparametric magnetic resonance imaging in recent years, the concerns surrounding the variability of imaging quality have garnered increased attention. Several factors contribute to the inconsistency of image quality, such as acquisition parameters, scanner differences and interobserver variabilities. While efforts have been made to standardize image acquisition and interpretation via the development of systems, such as PI-RADS and PI-QUAL, the scoring systems still depend on the subjective experience and acumen of humans. Artificial intelligence (AI) has been increasingly used in many applications, including medical imaging, due to its ability to automate tasks and lower human error rates. These advantages have the potential to standardize the tasks of image interpretation and quality control of prostate MRI. Despite its potential, thorough validation is required before the implementation of AI in clinical practice. In this article, we explore the opportunities and challenges of AI, with a focus on the interpretation and quality of prostate MRI.

Surface-GCN: Learning interaction experience for organ segmentation in 3D medical images

BACKGROUND

Accurate segmentation of organs has a great significance for clinical diagnosis, but it is still hard work due to the obscure imaging boundaries caused by tissue adhesion on medical images. Based on the image continuity in medical image volumes, segmentation on these slices could be inferred from adjacent slices with a clear organ boundary. Radiologists can delineate a clear organ boundary by observing adjacent slices.

PURPOSE

Inspired by the radiologists' delineating procedure, we design an organ segmentation model based on boundary information of adjacent slices and a human-machine interactive learning strategy to introduce clinical experience.

METHODS

We propose an interactive organ segmentation method for medical image volume based on Graph Convolution Network (GCN) called Surface-GCN. First, we propose a Surface Feature Extraction Network (SFE-Net) to capture surface features of a target organ, and supervise it by a Mini-batch Adaptive Surface Matching (MBASM) module. Then, to predict organ boundaries precisely, we design an automatic segmentation module based on a Surface Convolution Unit (SCU), which propagates information on organ surfaces to refine the generated boundaries. In addition, an interactive segmentation module is proposed to learn radiologists' experience of interactive corrections on organ surfaces to reduce interaction clicks.

RESULTS

We evaluate the proposed method on one prostate MR image dataset and two abdominal multi-organ CT datasets. The experimental results show that our method outperforms other state-of-the-art methods. For prostate segmentation, the proposed method conducts a DSC score of 94.49% on PROMISE12 test dataset. For abdominal multi-organ segmentation, the proposed method achieves DSC scores of 95, 91, 95, and 88% for the left kidney, gallbladder, spleen, and esophagus, respectively. For interactive segmentation, the proposed method reduces 5-10 interaction clicks to reach the same accuracy.

CONCLUSIONS

To overcome the medical organ segmentation challenge, we propose a Graph Convolutional Network called Surface-GCN by imitating radiologist interactions and learning clinical experience. On single and multiple organ segmentation tasks, the proposed method could obtain more accurate segmentation boundaries compared with other state-of-the-art methods.

Pulmonary nodules segmentation based on domain adaptation

With the development of deep learning, the methods based on transfer learning have promoted the progress of medical image segmentation. However, the domain shift and complex background information of medical images limit the further improvement of the segmentation accuracy. Domain adaptation can compensate for the sample shortage by learning important information from a similar source dataset. Therefore, a segmentation method based on adversarial domain adaptation with background mask (ADAB) is proposed in this paper. Firstly, two ADAB networks are built for the source and target data segmentation, respectively. Next, to extract the foreground features that are the input of the discriminators, the background masks are generated according to the region growth algorithm. Then, to update the parameters in the target network without being affected by the conflict between the distinguishing differences of the discriminator and the domain shift reduction of the adversarial domain adaptation, a gradient reversal layer propagation is embedded in the ADAB model for the target data. Finally, an enhanced boundaries loss is deduced to make the target network sensitive to the edge of the area to be segmented. The performance of the proposed method is evaluated in the segmentation of pulmonary nodules in computed tomography images. Experimental results show that the proposed approach has a potential prospect in medical image processing.

TBENet:A two-branch boundary enhancement Network for cerebrovascular segmentation

Cerebrovascular segmentation in digital subtraction angiography (DSA) images is the gold standard for clinical diagnosis. However, owing to the complexity of cerebrovascular, automatic cerebrovascular segmentation in DSA images is a challenging task. In this paper, we propose a CNN-based Two-branch Boundary Enhancement Network (TBENet) for automatic segmentation of cerebrovascular in DSA images. The TBENet is inspired by U-Net and designed as an encoder-decoder architecture. We propose an additional boundary branch to segment the boundary of cerebrovascular and a Main and Boundary branches Fusion Module (MBFM) to integrate the boundary branch outcome with the main branch outcome to achieve better segmentation performance. The TBENet was evaluated on HMCDSA (an in-house DSA cerebrovascular dataset), and reaches 0.9611, 0.7486, 0.7152, 0.9860 and 0.9556 in Accuracy, F1 score, Sensitivity, Specificity, and AUC, respectively. Meanwhile, we tested our TBENet on the public vessel segmentation benchmark DRIVE, and the results show that our TBENet can be extended to diverse vessel segmentation tasks.

CDRNet: Cascaded dense residual network for grayscale and pseudocolor medical image fusion

OBJECTIVE

Multimodal medical fusion images have been widely used in clinical medicine, computer-aided diagnosis and other fields. However, the existing multimodal medical image fusion algorithms generally have shortcomings such as complex calculations, blurred details and poor adaptability. To solve this problem, we propose a cascaded dense residual network and use it for grayscale and pseudocolor medical image fusion.

METHODS

The cascaded dense residual network uses a multiscale dense network and a residual network as the basic network architecture, and a multilevel converged network is obtained through cascade. The cascaded dense residual network contains 3 networks, the first-level network inputs two images with different modalities to obtain a fused Image 1, the second-level network uses fused Image 1 as the input image to obtain fused Image 2 and the third-level network uses fused Image 2 as the input image to obtain fused Image 3. The multimodal medical image is trained through each level of the network, and the output fusion image is enhanced step-by-step.

RESULTS

As the number of networks increases, the fusion image becomes increasingly clearer. Through numerous fusion experiments, the fused images of the proposed algorithm have higher edge strength, richer details, and better performance in the objective indicators than the reference algorithms.

CONCLUSION

Compared with the reference algorithms, the proposed algorithm has better original information, higher edge strength, richer details and an improvement of the four objective SF, AG, MZ and EN indicator metrics.

Dynamic Cross-Task Representation Adaptation for Clinical Targets Co-Segmentation in CT Image-Guided Post-Prostatectomy Radiotherapy

Adjuvant and salvage radiotherapy after radical prostatectomy requires precise delineations of prostate bed (PB), i.e., the clinical target volume, and surrounding organs at risk (OARs) to optimize radiotherapy planning. Segmenting PB is particularly challenging even for clinicians, e.g., from the planning computed tomography (CT) images, as it is an invisible/virtual target after the operative removal of the cancerous prostate gland. Very recently, a few deep learning-based methods have been proposed to automatically contour non-contrast PB by leveraging its spatial reliance on adjacent OARs (i.e., the bladder and rectum) with much more clear boundaries, mimicking the clinical workflow of experienced clinicians. Although achieving state-of-the-art results from both the clinical and technical aspects, these existing methods improperly ignore the gap between the hierarchical feature representations needed for segmenting those fundamentally different clinical targets (i.e., PB and OARs), which in turn limits their delineation accuracy. This paper proposes an asymmetric multi-task network integrating dynamic cross-task representation adaptation (i.e., DyAdapt) for accurate and efficient co-segmentation of PB and OARs in one-pass from CT images. In the learning-to-learn framework, the DyAdapt modules adaptively transfer the hierarchical feature representations from the source task of OARs segmentation to match up with the target (and more challenging) task of PB segmentation, conditioned on the dynamic inter-task associations learned from the learning states of the feed-forward path. On a real-patient dataset, our method led to state-of-the-art results of PB and OARs co-segmentation. Code is available at https://github.com/ladderlab-xjtu/DyAdapt.

Deep Learning-Based Auto-Segmentation of Spinal Cord Internal Structure of Diffusion Tensor Imaging in Cervical Spondylotic Myelopathy

Cervical spondylotic myelopathy (CSM) is a chronic disorder of the spinal cord. ROI-based features on diffusion tensor imaging (DTI) provide additional information about spinal cord status, which would benefit the diagnosis and prognosis of CSM. However, the manual extraction of the DTI-related features on multiple ROIs is time-consuming and laborious. In total, 1159 slices at cervical levels from 89 CSM patients were analyzed, and corresponding fractional anisotropy (FA) maps were calculated. Eight ROIs were drawn, covering both sides of lateral, dorsal, ventral, and gray matter. The UNet model was trained with the proposed heatmap distance loss for auto-segmentation. Mean Dice coefficients on the test dataset for dorsal, lateral, and ventral column and gray matter were 0.69, 0.67, 0.57, 0.54 on the left side and 0.68, 0.67, 0.59, 0.55 on the right side. The ROI-based mean FA value based on segmentation model strongly correlated with the value based on manual drawing. The percentages of the mean absolute error between the two values of multiple ROIs were 0.07, 0.07, 0.11, and 0.08 on the left side and 0.07, 0.1, 0.1, 0.11, and 0.07 on the right side. The proposed segmentation model has the potential to offer a more detailed spinal cord segmentation and would be beneficial for quantifying a more detailed status of the cervical spinal cord.

Boundary-Sensitive Loss Function With Location Constraint for Hard Region Segmentation

In computer-aided diagnosis and treatment planning, accurate segmentation of medical images plays an essential role, especially for some hard regions including boundaries, small objects and background interference. However, existing segmentation loss functions including distribution-, region- and boundary-based losses cannot achieve satisfactory performances on these hard regions. In this paper, a boundary-sensitive loss function with location constraint is proposed for hard region segmentation in medical images, which provides three advantages: i) our Boundary-Sensitive loss (BS-loss) can automatically pay more attention to the hard-to-segment boundaries (e.g., thin structures and blurred boundaries), thus obtaining finer object boundaries; ii) BS-loss also can adjust its attention to small objects during training to segment them more accurately; and iii) our location constraint can alleviate the negative impact of the background interference, through the distribution matching of pixels between prediction and Ground Truth (GT) along each axis. By resorting to the proposed BS-loss and location constraint, the hard regions in both foreground and background are considered. Experimental results on three public datasets demonstrate the superiority of our method. Specifically, compared to the second-best method tested in this study, our method improves performance on hard regions in terms of Dice similarity coefficient (DSC) and 95% Hausdorff distance (95%HD) of up to 4.17% and 73% respectively. In addition, it also achieves the best overall segmentation performance. Hence, we can conclude that our method can accurately segment these hard regions and improve the overall segmentation performance in medical images.

A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning

This survey aims to identify commonly used methods, datasets, future trends, knowledge gaps, constraints, and limitations in the field to provide an overview of current solutions used in medical image analysis in parallel with the rapid developments in transfer learning (TL). Unlike previous studies, this survey grouped the last five years of current studies for the period between January 2017 and February 2021 according to different anatomical regions and detailed the modality, medical task, TL method, source data, target data, and public or private datasets used in medical imaging. Also, it provides readers with detailed information on technical challenges, opportunities, and future research trends. In this way, an overview of recent developments is provided to help researchers to select the most effective and efficient methods and access widely used and publicly available medical datasets, research gaps, and limitations of the available literature.

Automatic segmentation of prostate MRI based on 3D pyramid pooling Unet

PURPOSE

Automatic segmentation of prostate magnetic resonance (MR) images is crucial for the diagnosis, evaluation, and prognosis of prostate diseases (including prostate cancer). In recent years, the mainstream segmentation method for the prostate has been converted to convolutional neural networks. However, owing to the complexity of the tissue structure in MR images and the limitations of existing methods in spatial context modeling, the segmentation performance should be improved further.

METHODS

In this study, we proposed a novel 3D pyramid pool Unet that benefits from the pyramid pooling structure embedded in the skip connection (SC) and the deep supervision (DS) in the up-sampling of the 3D Unet. The parallel SC of the conventional 3D Unet network causes low-resolution information to be sent to the feature map repeatedly, resulting in blurred image features. To overcome the shortcomings of the conventional 3D Unet, we merge each decoder layer with the feature map of the same scale as the encoder and the smaller scale feature map of the pyramid pooling encoder. This SC combines the low-level details and high-level semantics at two different levels of feature maps. In addition, pyramid pooling performs multifaceted feature extraction on each image behind the convolutional layer, and DS learns hierarchical representations from comprehensive aggregated feature maps, which can improve the accuracy of the task.

RESULTS

Experiments on 3D prostate MR images of 78 patients demonstrated that our results were highly correlated with expert manual segmentation. The average relative volume difference and Dice similarity coefficient of the prostate volume area were 2.32% and 91.03%, respectively.

CONCLUSION

Quantitative experiments demonstrate that, compared with other methods, the results of our method are highly consistent with the expert manual segmentation.

Deep learning on lateral flow immunoassay for the analysis of detection data

Lateral flow immunoassay (LFIA) is an important detection method in vitro diagnosis, which has been widely used in medical industry. It is difficult to analyze all peak shapes through classical methods due to the complexity of LFIA. Classical methods are generally some peak-finding methods, which cannot distinguish the difference between normal peak and interference or noise peak, and it is also difficult for them to find the weak peak. Here, a novel method based on deep learning was proposed, which can effectively solve these problems. The method had two steps. The first was to classify the data by a classification model and screen out double-peaks data, and second was to realize segmentation of the integral regions through an improved U-Net segmentation model. After training, the accuracy of the classification model for validation set was 99.59%, and using combined loss function (WBCE + DSC), intersection over union (IoU) value of segmentation model for validation set was 0.9680. This method was used in a hand-held fluorescence immunochromatography analyzer designed independently by our team. A Ferritin standard curve was created, and the T/C value correlated well with standard concentrations in the range of 0-500 ng/ml (R ² = 0.9986). The coefficients of variation (CVs) were ≤ 1.37%. The recovery rate ranged from 96.37 to 105.07%. Interference or noise peaks are the biggest obstacle in the use of hand-held instruments, and often lead to peak-finding errors. Due to the changeable and flexible use environment of hand-held devices, it is not convenient to provide any technical support. This method greatly reduced the failure rate of peak finding, which can reduce the customer's need for instrument technical support. This study provided a new direction for the data-processing of point-of-care testing (POCT) instruments based on LFIA.

A singular Riemannian geometry approach to deep neural networks II. Reconstruction of 1-D equivalence classes

We proposed in a previous work a geometric framework to study a deep neural network, seen as sequence of maps between manifolds, employing singular Riemannian geometry. In this paper, we present an application of this framework, proposing a way to build the class of equivalence of an input point: such class is defined as the set of the points on the input manifold mapped to the same output by the neural network. In other words, we build the preimage of a point in the output manifold in the input space. In particular. We focus for simplicity on the case of neural networks maps from n-dimensional real spaces to (n-1)-dimensional real spaces, we propose an algorithm allowing to build the set of points lying on the same class of equivalence. This approach leads to two main applications: the generation of new synthetic data and it may provides some insights on how a classifier can be confused by small perturbation on the input data (e.g. a penguin image classified as an image containing a chihuahua). In addition, for neural networks from 2D to 1D real spaces, we also discuss how to find the preimages of closed intervals of the real line. We also present some numerical experiments with several neural networks trained to perform non-linear regression tasks, including the case of a binary classifier.

CellTranspose: Few-shot Domain Adaptation for Cellular Instance Segmentation

Automated cellular instance segmentation is a process utilized for accelerating biological research for the past two decades, and recent advancements have produced higher quality results with less effort from the biologist. Most current endeavors focus on completely cutting the researcher out of the picture by generating highly generalized models. However, these models invariably fail when faced with novel data, distributed differently than the ones used for training. Rather than approaching the problem with methods that presume the availability of large amounts of target data and computing power for retraining, in this work we address the even greater challenge of designing an approach that requires minimal amounts of new annotated data as well as training time. We do so by designing specialized contrastive losses that leverage the few annotated samples very efficiently. A large set of results show that 3 to 5 annotations lead to models with accuracy that: 1) significantly mitigate the covariate shift effects; 2) matches or surpasses other adaptation methods; 3) even approaches methods that have been fully retrained on the target distribution. The adaptation training is only a few minutes, paving a path towards a balance between model performance, computing requirements and expert-level annotation needs.

An explainable deep learning-based algorithm with an attention mechanism for predicting the live birth potential of mouse embryos

In assisted reproductive technology (ART), embryos produced by in vitro fertilization (IVF) are graded according to their live birth potential, and high-grade embryos are preferentially transplanted. However, rates of live birth following clinical ART remain low worldwide. Grading is based on the embryo shape at a limited number of stages and does not consider the shape of embryos and intracellular structures, e.g., nuclei, at various stages important for normal embryogenesis. Here, we developed a Normalized Multi-View Attention Network (NVAN) that directly predicts live birth potential from the nuclear structure in live-cell fluorescence images of mouse embryos from zygote to across a wide range of stages. The input is morphological features of cell nuclei, which were extracted as multivariate time-series data by using the segmentation algorithm for mouse embryos. The classification accuracy of our method (83.87%) greatly exceeded that of existing machine-learning methods and that of visual inspection by embryo culture specialists. Our method also has a new attention mechanism that allows us to determine which values of multivariate time-series data, used to describe nuclear morphology, were the basis for the prediction. By visualizing the features that contributed most to the prediction of live birth potential, we found that the size and shape of the nucleus at the morula stage and at the time of cell division were important for live birth prediction. We anticipate that our method will help ART and developmental engineering as a new basic technology for IVF embryo selection.

FBCU-Net: A fine-grained context modeling network using boundary semantic features for medical image segmentation

The performance of deep learning-based medical image segmentation methods largely depends on the segmentation accuracy of tissue boundaries. However, since the boundary region is at the junction of areas of different categories, the pixels located at the boundary inevitably carry features belonging to other classes and difficult to distinguish. This paper proposes a fine-grained contextual modeling network for medical image segmentation based on boundary semantic features, FBCU-Net, which uses the semantic features of boundary regions to reduce the influence of irrelevant features on boundary pixels. First, based on the discovery that indistinguishable pixels are usually boundary pixels in medical images, we introduce new supervision information to find and classify boundary pixels. Second, based on the existing relational context modeling schemes, we generate the boundary region representations representing the semantic features of boundary regions. Last, we use boundary region representations to reduce the influence of irrelevant features on boundary pixels and generate highly discriminative pixel representations. Furthermore, to enhance the attention of the network to the boundary region, we also propose the boundary enhancement strategy. We evaluate the proposed model on five datasets, TUI (Thyroid Tumor), ISIC-2018 (Dermoscopy), 2018 Data Science Bowl (Cell Nuclei), Glas (Colon Cancer), and BUSI (Breast Cancer). The results show that FBCU-Net has better boundary segmentation performance and overall performance for different medical images than other state-of-the-art (SOTA) methods, and has great potential for clinical application.

A novel MCF-Net: Multi-level context fusion network for 2D medical image segmentation

Medical image segmentation is a crucial step in the clinical applications for diagnosis and analysis of some diseases. U-Net-based convolution neural networks have achieved impressive performance in medical image segmentation tasks. However, the multi-level contextual information integration capability and the feature extraction ability are often insufficient. In this paper, we present a novel multi-level context fusion network (MCF-Net) to improve the performance of U-Net on various segmentation tasks by designing three modules, hybrid attention-based residual atrous convolution (HARA) module, multi-scale feature memory (MSFM) module, and multi-receptive field fusion (MRFF) module, to fuse multi-scale contextual information. HARA module was proposed to effectively extract multi-receptive field features by combing atrous spatial pyramid pooling and attention mechanism. We further design the MSFM and MRFF modules to fuse features of different levels and effectively extract contextual information. The proposed MCF-Net was evaluated on the ISIC 2018, DRIVE, BUSI, and Kvasir-SEG datasets, which have challenging images of many sizes and widely varying anatomy. The experimental results show that MCF-Net is very competitive with other U-Net models, and it offers tremendous potential as a general-purpose deep learning model for 2D medical image segmentation.

Multi-scale discriminative network for prostate cancer lesion segmentation in multiparametric MR images

PURPOSE

The accurate and reliable segmentation of prostate cancer (PCa) lesions using multiparametric magnetic resonance imaging (mpMRI) sequences, is crucial to the image-guided intervention and treatment of prostate disease. For PCa lesion segmentation, it is essential to reliably combine local and global information to retain the features of small targets at multiple scales. Therefore, this study proposes a multi-scale segmentation network with a cascading pyramid convolution module (CPCM) and a double-input channel attention module (DCAM) for the automated and accurate segmentation of PCa lesions using mpMRI.

METHODS

First, the region of interest was extracted from the data by clipping to enlarge the target region and reduce the background noise interference. Next, four CPCMs with large convolution kernels in their skip connection paths were designed to improve the feature extraction capability of the network for small targets. At the same time, a convolution decomposition was applied to reduce the computational complexity. Finally, the DCAM was adopted in the decoder to provide bottom-up semantic discriminative guidance; it can use the semantic information of the network's deep features to guide the shallow output of features with a higher discriminant ability. A residual refinement module (RRM) was also designed to strengthen the recognition ability of each stage. The feature maps of the skip connection and the decoder all go through the RRM.

RESULTS

For the Initiative for Collaborative Computer Vision Benchmarking (I2CVB) dataset, our proposed model achieved a Dice similarity coefficient (DSC) of 79.31% and an average boundary distance (ABD) of 4.15 mm. For the Prostate Multiparametric MRI (PROMM) dataset, our method greatly improved the DSC to 82.11% and obtained an ABD of 3.64 mm.

CONCLUSIONS

The experimental results of two different mpMRI prostate datasets demonstrate that our model is more accurate and reliable on small targets. In addition, it outperforms other state-of-the-art methods.

Grayscale self-adjusting network with weak feature enhancement for 3D lumbar anatomy segmentation

The automatic segmentation of lumbar anatomy is a fundamental problem for the diagnosis and treatment of lumbar disease. The recent development of deep learning techniques has led to remarkable progress in this task, including the possible segmentation of nerve roots, intervertebral discs, and dural sac in a single step. Despite these advances, lumbar anatomy segmentation remains a challenging problem due to the weak contrast and noise of input images, as well as the variability of intensities and size in lumbar structures across different subjects. To overcome these challenges, we propose a coarse-to-fine deep neural network framework for lumbar anatomy segmentation, which obtains a more accurate segmentation using two strategies. First, a progressive refinement process is employed to correct low-confidence regions by enhancing the feature representation in these regions. Second, a grayscale self-adjusting network (GSA-Net) is proposed to optimize the distribution of intensities dynamically. Experiments on datasets comprised of 3D computed tomography (CT) and magnetic resonance (MR) images show the advantage of our method over current segmentation approaches and its potential for diagnosing and lumbar disease treatment.

Automatic prostate and peri-prostatic fat segmentation based on pyramid mechanism fusion network for T2-weighted MRI

BACKGROUND AND OBJECTIVE

Automatic and accurate segmentation of prostate and peri-prostatic fat in male pelvic MRI images is a critical step in the diagnosis and prognosis of prostate cancer. The boundary of prostate tissue is not clear, which makes the task of automatic segmentation very challenging. The main issues, especially for the peri-prostatic fat, which is being offered for the first time, are hazy boundaries and a large form variation.

METHODS

We propose a pyramid mechanism fusion network (PMF-Net) to learn global features and more comprehensive context information. In the proposed PMF-Net, we devised two pyramid techniques in particular. A pyramid mechanism module made of dilated convolutions of varying rates is inserted before each down sample of the fundamental network architecture encoder. The module is intended to address the issue of information loss during the feature coding process, particularly in the case of segmentation object boundary information. In the transition stage from encoder to decoder, pyramid fusion module is designed to extract global features. The features of the decoder not only integrate the features of the previous stage after up sampling and the output features of pyramid mechanism, but also include the features of skipping connection transmission under the same scale of the encoder.

RESULTS

The segmentation results of prostate and peri-prostatic fat on numerous diverse male pelvic MRI datasets show that our proposed PMF-Net has higher performance than existing methods. The average surface distance (ASD) and Dice similarity coefficient (DSC) of prostate segmentation results reached 10.06 and 90.21%, respectively. The ASD and DSC of the peri-prostatic fat segmentation results reached 50.96 and 82.41%.

CONCLUSIONS

The results of our segmentation are substantially connected and consistent with those of expert manual segmentation. Furthermore, peri-prostatic fat segmentation is a new issue, and good automatic segmentation has substantial therapeutic implications.

Edge-preserving Image Synthesis for Unsupervised Domain Adaptation in Medical Image Segmentation

Domain Adaptation is a technique to address the lack of massive amounts of labeled data in different application domains. Unsupervised domain adaptation is the process of adapting a model to an unseen target dataset using solely labeled source data and unlabeled target domain data. Though many image-spaces domain adaptation methods have been proposed to capture pixel-level domain-shift, such techniques may fail to maintain high-level semantic information for the segmentation task. For the case of biomedical images, fine details such as blood vessels can be lost during the image transformation operations between domains. In this work, we propose a model that adapts between domains using cycle-consistent loss while maintaining edge details of the original images by enforcing an edge-based loss during the adaptation process. We demonstrate the effectiveness of our algorithm by comparing it to other approaches on two eye fundus vessels segmentation datasets. We achieve 3.1 % increment in Dice score compared to the SOTA and  ∼  7.02% increment compared to a vanilla CycleGAN implementation. Clinical relevance- The proposed adaptation scheme can provide better performance on unseen data for semantic segmentation, which is widely applied in computer-aided diagnosis. Such robust performance can reduce the reliance of a large amount of labeled data, which is a common problem in the medical domain.

ACE R-CNN: An Attention Complementary and Edge Detection-Based Instance Segmentation Algorithm for Individual Tree Species Identification Using UAV RGB Images and LiDAR Data

Accurate and automatic identification of tree species information at the individual tree scale is of great significance for fine-scale investigation and management of forest resources and scientific assessment of forest ecosystems. Despite the fact that numerous studies have been conducted on the delineation of individual tree crown and species classification using drone high-resolution red, green and blue (RGB) images, and Light Detection and Ranging (LiDAR) data, performing the above tasks simultaneously has rarely been explored, especially in complex forest environments. In this study, we improve upon the state of the Mask region-based convolution neural network (Mask R-CNN) with our proposed attention complementary network (ACNet) and edge detection R-CNN (ACE R-CNN) for individual tree species identification in high-density and complex forest environments. First, we propose ACNet as the feature extraction backbone network to fuse the weighted features extracted from RGB images and canopy height model (CHM) data through an attention complementary module, which is able to selectively fuse weighted features extracted from RGB and CHM data at different scales, and enables the network to focus on more effective information. Second, edge loss is added to the loss function to improve the edge accuracy of the segmentation, which is calculated through the edge detection filter introduced in the Mask branch of Mask R-CNN. We demonstrate the performance of ACE R-CNN for individual tree species identification in three experimental areas of different tree species in southern China with precision (P), recall (R), F1-score, and average precision (AP) above 0.9. Our proposed ACNet–the backbone network for feature extraction–has better performance in individual tree species identification compared with the ResNet50-FPN (feature pyramid network). The addition of the edge loss obtained by the Sobel filter further improves the identification accuracy of individual tree species and accelerates the convergence speed of the model training. This work demonstrates the improved performance of ACE R-CNN for individual tree species identification and provides a new solution for tree-level species identification in complex forest environments, which can support carbon stock estimation and biodiversity assessment.

Shadow-Consistent Semi-Supervised Learning for Prostate Ultrasound Segmentation

Prostate segmentation in transrectal ultrasound (TRUS) image is an essential prerequisite for many prostate-related clinical procedures, which, however, is also a long-standing problem due to the challenges caused by the low image quality and shadow artifacts. In this paper, we propose a Shadow-consistent Semi-supervised Learning (SCO-SSL) method with two novel mechanisms, namely shadow augmentation (Shadow-AUG) and shadow dropout (Shadow-DROP), to tackle this challenging problem. Specifically, Shadow-AUG enriches training samples by adding simulated shadow artifacts to the images to make the network robust to the shadow patterns. Shadow-DROP enforces the segmentation network to infer the prostate boundary using the neighboring shadow-free pixels. Extensive experiments are conducted on two large clinical datasets (a public dataset containing 1,761 TRUS volumes and an in-house dataset containing 662 TRUS volumes). In the fully-supervised setting, a vanilla U-Net equipped with our Shadow-AUG&Shadow-DROP outperforms the state-of-the-arts with statistical significance. In the semi-supervised setting, even with only 20% labeled training data, our SCO-SSL method still achieves highly competitive performance, suggesting great clinical value in relieving the labor of data annotation. Source code is released at https://github.com/DIAL-RPI/SCO-SSL.

Integration of Deep Learning and Active Shape Models for More Accurate Prostate Segmentation in 3D MR Images

Magnetic resonance imaging (MRI) has a growing role in the clinical workup of prostate cancer. However, manual three-dimensional (3D) segmentation of the prostate is a laborious and time-consuming task. In this scenario, the use of automated algorithms for prostate segmentation allows us to bypass the huge workload of physicians. In this work, we propose a fully automated hybrid approach for prostate gland segmentation in MR images using an initial segmentation of prostate volumes using a custom-made 3D deep network (VNet-T2), followed by refinement using an Active Shape Model (ASM). While the deep network focuses on three-dimensional spatial coherence of the shape, the ASM relies on local image information and this joint effort allows for improved segmentation of the organ contours. Our method is developed and tested on a dataset composed of T2-weighted (T2w) MRI prostatic volumes of 60 male patients. In the test set, the proposed method shows excellent segmentation performance, achieving a mean dice score and Hausdorff distance of 0.851 and 7.55 mm, respectively. In the future, this algorithm could serve as an enabling technology for the development of computer-aided systems for prostate cancer characterization in MR imaging.

Polar transform network for prostate ultrasound segmentation with uncertainty estimation

Automatic and accurate prostate ultrasound segmentation is a long-standing and challenging problem due to the severe noise and ambiguous/missing prostate boundaries. In this work, we propose a novel polar transform network (PTN) to handle this problem from a fundamentally new perspective, where the prostate is represented and segmented in the polar coordinate space rather than the original image grid space. This new representation gives a prostate volume, especially the most challenging apex and base sub-areas, much denser samples than the background and thus facilitate the learning of discriminative features for accurate prostate segmentation. Moreover, in the polar representation, the prostate surface can be efficiently parameterized using a 2D surface radius map with respect to a centroid coordinate, which allows the proposed PTN to obtain superior accuracy compared with its counterparts using convolutional neural networks while having significantly fewer (18%∼41%) trainable parameters. We also equip our PTN with a novel strategy of centroid perturbed test-time augmentation (CPTTA), which is designed to further improve the segmentation accuracy and quantitatively assess the model uncertainty at the same time. The uncertainty estimation function provides valuable feedback to clinicians when manual modifications or approvals are required for the segmentation, substantially improving the clinical significance of our work. We conduct a three-fold cross validation on a clinical dataset consisting of 315 transrectal ultrasound (TRUS) images to comprehensively evaluate the performance of the proposed method. The experimental results show that our proposed PTN with CPTTA outperforms the state-of-the-art methods with statistical significance on most of the metrics while exhibiting a much smaller model size. Source code of the proposed PTN is released at https://github.com/DIAL-RPI/PTN.

Unsupervised domain adaptation with hyperbolic graph convolution network for segmentation of X-ray breast mass

Deep learning has been widely used in medical image segmentation, such as breast tumor segmentation, prostate MR image segmentation, and so on. However, the labeling of the data set takes a lot of time. Although the emergence of unsupervised domain adaptation fills the technical gap, the existing domain adaptation methods for breast segmentation do not consider the alignment of the source domain and target domain breast mass structure. This paper proposes a hyperbolic graph convolutional network architecture. First, a hyperbolic graph convolutional network is used to make the source and target domains structurally aligned. Secondly, we adopt a hyperbolic space mapping model that has better expressive ability than Euclidean space in a graph structure. In particular, when constructing the graph structure, we added the completion adjacency matrix, so that the graph structure can be changed after each feature mapping, which can better improve the segmentation accuracy. Extensive comparative and ablation experiments were performed on two common breast datasets(CBIS-DDSM and INbreast). Experiments show that the method in this paper is better than the most advanced model. When CBIS-DDSM and INbreast are used as the source domain, the segmentation accuracy reaches 89.1% and 80.7%.

Domain Adaptation for Medical Image Analysis: A Survey

Machine learning techniques used in computer-aided medical image analysis usually suffer from the domain shift problem caused by different distributions between source/reference data and target data. As a promising solution, domain adaptation has attracted considerable attention in recent years. The aim of this paper is to survey the recent advances of domain adaptation methods in medical image analysis. We first present the motivation of introducing domain adaptation techniques to tackle domain heterogeneity issues for medical image analysis. Then we provide a review of recent domain adaptation models in various medical image analysis tasks. We categorize the existing methods into shallow and deep models, and each of them is further divided into supervised, semi-supervised and unsupervised methods. We also provide a brief summary of the benchmark medical image datasets that support current domain adaptation research. This survey will enable researchers to gain a better understanding of the current status, challenges and future directions of this energetic research field.

Multi-Species Individual Tree Segmentation and Identification Based on Improved Mask R-CNN and UAV Imagery in Mixed Forests

High-resolution UAV imagery paired with a convolutional neural network approach offers significant advantages in accurately measuring forestry ecosystems. Despite numerous studies existing for individual tree crown delineation, species classification, and quantity detection, the comprehensive situation in performing the above tasks simultaneously has rarely been explored, especially in mixed forests. In this study, we propose a new method for individual tree segmentation and identification based on the improved Mask R-CNN. For the optimized network, the fusion type in the feature pyramid network is modified from down-top to top-down to shorten the feature acquisition path among the different levels. Meanwhile, a boundary-weighted loss module is introduced to the cross-entropy loss function Lmask to refine the target loss. All geometric parameters (contour, the center of gravity and area) associated with canopies ultimately are extracted from the mask by a boundary segmentation algorithm. The results showed that F1-score and mAP for coniferous species were higher than 90%, and that of broadleaf species were located between 75%–85.44%. The producer’s accuracy of coniferous forests was distributed between 0.8–0.95 and that of broadleaf ranged in 0.87–0.93; user’s accuracy of coniferous was distributed between 0.81–0.84 and that of broadleaf ranged in 0.71–0.76. The total number of trees predicted was 50,041 for the entire study area, with an overall error of 5.11%. The method under study is compared with other networks including U-net and YOLOv3. Results in this study show that the improved Mask R-CNN has more advantages in broadleaf canopy segmentation and number detection.

Automatic Dental Plaque Segmentation based on Local-to-global Features Fused Self-attention Network

The accurate detection of dental plaque at an early stage will definitely prevent periodontal diseases and dental caries. However, it remains difficult for the current dental examination to accurately recognize dental plague without using medical dyeing reagent due to the low contrast between dental plaque and healthy teeth. To combat this problem, this paper proposes a novel network enhanced by a self-attention module for intelligent dental plaque segmentation. The key motivation is to directly utilize oral endoscope images (bypassing the need of dyeing reagent) and get the accurate pixel-level dental plaque segmentation results. The algorithm needs to conduct self-attention at the super-pixel level and fuse the super-pixels' local-to-global features. Our newly-designed network architecture will afford the simultaneous fusion of multiple-scale complementary information guided by the powerful deep learning paradigm. The critical fused information includes the statistical distribution of the plaques color, the heat kernel signature (HKS) based local-to-global structure relationship, and the circle-LBP based local texture pattern in the nearby regions centering around the plaque area. To further refine the fuzed multiple-scale features, we devise an attention module based on CNN, which could focalize the regions of interest in plaque more easily, especially for much challenging cases. Extensive experiments and comprehensive evaluations confirm that, for a small-scale training dataset, our method could outperform the state-of-the-art methods. Meanwhile, the user studies verify the claim that our method is more accurate than conventional dental practice conducted by experienced dentists.