Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation

Predicting the efficacy of microwave ablation of benign thyroid nodules from ultrasound images using deep convolutional neural networks

BACKGROUND

Thyroid nodules are frequent in clinical settings, and their diagnosis in adults is growing, with some persons experiencing symptoms. Ultrasound-guided thermal ablation can shrink nodules and alleviate discomfort. Because the degree and rate of lesion absorption vary greatly between individuals, there is no reliable model for predicting the therapeutic efficacy of thermal ablation.

METHODS

Five convolutional neural network models including VGG19, Resnet 50, EfficientNetB1, EfficientNetB0, and InceptionV3, pre-trained with ImageNet, were compared for predicting the efficacy of ultrasound-guided microwave ablation (MWA) for benign thyroid nodules using ultrasound data. The patients were randomly assigned to one of two data sets: training (70%) or validation (30%). Accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and area under the curve (AUC) were all used to assess predictive performance.

RESULTS

In the validation set, fine-tuned EfficientNetB1 performed best, with an AUC of 0.85 and an ACC of 0.79.

CONCLUSIONS

The study found that our deep learning model accurately predicts nodules with VRR < 50% after a single MWA session. Indeed, when thermal therapies compete with surgery, anticipating which nodules will be poor responders provides useful information that may assist physicians and patients determine whether thermal ablation or surgery is the preferable option. This was a preliminary study of deep learning, with a gap in actual clinical applications. As a result, more in-depth study should be undertaken to develop deep-learning models that can better help clinics. Prospective studies are expected to generate high-quality evidence and improve clinical performance in subsequent research.

Diffusion equation quantification: selective enhancement algorithm for bone metastasis lesions in CT images

Objective.Diffusion equation (DE) imaging processing is promising to enhance images showing lesions of bone metastasis (LBM). The Perona-Malik diffusion (PMD) model, which has been widely used and studied, is an anisotropic diffusion processing method to denoise or extract objects from an image effectively. However, the smoothing characteristics of PMD or its related method hinder extraction and enhancement of soft tissue regions of medical image such as computed tomography (CT), typically leaving an indistinct region with ambient tissues. Moreover, PMD expands the border region of the objects. A novel diffusion methodology must be used to enhance the LBM region effectively.Approach.For this study, we originally developed a DE quantification (DEQ) method that uses a filter function to selectively provide modulated diffusion according to the original locations of objects in an image. The structural similarity index measure (SSIM) and Lie derivative image analysisL-value map were used to evaluate image quality and processing.Main results.We determined superellipse function with its ordern=4as a better performing filter for the LBM region. DEQ was found to be more effective at contrasting LBM for various LBM CT images than PMD or its improved models when the filter was a positive exponential similar function. DEQ yields enhancement agreeing with the indications of positron emission tomography despite complex LBM comprising osteoblastic, osteoclastic, mixed tissues, and metal artifacts, which is innovative. Moreover, DEQ retained high quality of image (SSIM> 0.95), and achieved a low mean value of theL-value (<0.001), indicative of our intended selective diffusion compared to other PMD models.Significance.Our method improved the visibility of mixed tissue lesions, which can assist computer visional framework and can help radiologists to produce accurate diagnose of LBM regions which are frequently overlooked in radiology findings because of the various degrees of visibility in CT images.

Artificial intelligence and deep learning algorithms for epigenetic sequence analysis: A review for epigeneticists and AI experts

Epigenetics encompasses mechanisms that can alter the expression of genes without changing the underlying genetic sequence. The epigenetic regulation of gene expression is initiated and sustained by several mechanisms such as DNA methylation, histone modifications, chromatin conformation, and non-coding RNA. The changes in gene regulation and expression can manifest in the form of various diseases and disorders such as cancer and congenital deformities. Over the last few decades, high-throughput experimental approaches have been used to identify and understand epigenetic changes, but these laboratory experimental approaches and biochemical processes are time-consuming and expensive. To overcome these challenges, machine learning and artificial intelligence (AI) approaches have been extensively used for mapping epigenetic modifications to their phenotypic manifestations. In this paper we provide a narrative review of published research on AI models trained on epigenomic data to address a variety of problems such as prediction of disease markers, gene expression, enhancer-promoter interaction, and chromatin states. The purpose of this review is twofold as it is addressed to both AI experts and epigeneticists. For AI researchers, we provided a taxonomy of epigenetics research problems that can benefit from an AI-based approach. For epigeneticists, given each of the above problems we provide a list of candidate AI solutions in the literature. We have also identified several gaps in the literature, research challenges, and recommendations to address these challenges.

Time-frequency-space transformer EEG decoding for spinal cord injury

Transformer neural networks based on multi-head self-attention are effective in several fields. To capture brain activity on electroencephalographic (EEG) signals and construct an effective pattern recognition model, this paper explores the multi-channel deep feature decoding method utilizing the self-attention mechanism. By integrating inter-channel features with intra-channel features, the self-attention mechanism generates a deep feature vector that encompasses information from all brain activities. In this paper, a time-frequency-spatial domain analysis of motor imagery (MI) based EEG signals from spinal cord injury patients is performed to construct a transformer neural network-based MI classification model. The proposed algorithm is named time-frequency-spatial transformer. The time-frequency and spatial domain feature vectors extracted from the EEG signals are input into the transformer neural network for multiple self-attention depth feature encoding, a peak classification accuracy of 93.56% is attained through the fully connected layer. By constructing the attention matrix brain network, it can be inferred that the channel connections constructed by the attention heads have similarities to the brain networks constructed by the EEG raw signals. The experimental results reveal that the self-attention coefficient brain network holds significant potential for brain activity analysis. The self-attention coefficient brain network can better illustrate correlated connections and show sample differences. Attention coefficient brain networks can provide a more discriminative approach for analyzing brain activity in clinical settings.

Exploring the efficacy of various CNN architectures in diagnosing oral cancer from squamous cell carcinoma

Oral cancer can result from mutations in cells located in the lips or mouth. Diagnosing oral cavity squamous cell carcinoma (OCSCC) is particularly challenging, often occurring at advanced stages. To address this, computer-aided diagnosis methods are increasingly being used. In this work, a deep learning-based approach utilizing models such as VGG16, ResNet50, LeNet-5, MobileNetV2, and Inception V3 is presented. NEOR and OCSCC datasets were used for feature extraction, with virtual slide images divided into tiles and classified as normal or squamous cell cancer. Performance metrics like accuracy, F1-score, AUC, precision, and recall were analyzed to determine the prerequisites for optimal CNN performance. The proposed CNN approaches were effective for classifying OCSCC and oral dysplasia, with the highest accuracy of 95.41 % achieved using MobileNetV2.

Key findings

Deep learning models, particularly MobileNetV2, achieved high classification accuracy (95.41 %) for OCSCC.CNN-based methods show promise for early-stage OCSCC and oral dysplasia diagnosis. Performance parameters like precision, recall, and F1-score help optimize CNN model selection for this task.

Leveraging 3D Convolutional Neural Networks for Accurate Recognition and Localization of Ankle Fractures

Background

Ankle fractures are common injuries with substantial implications for patient mobility and quality of life. Traditional imaging methods, while standard, have limitations in detecting subtle fractures and distinguishing them from complex bone structures. The advent of 3D Convolutional Neural Networks (3D-CNNs) offers a promising avenue for enhancing the accuracy and reliability of ankle fracture diagnoses.

Methods

In this study, we acquired 1453 high-resolution CT scans and processed them through three distinct 3D-CNN models: 3D-Mobilenet, 3D-Resnet101, and 3D-EfficientNetB7. Our approach involved meticulous preprocessing of images, including normalization and resampling, followed by a systematic comparative evaluation of the models based on accuracy, Area Under the Curve (AUC), and recall metrics. Additionally, the integration of Gradient-weighted Class Activation Mapping (Grad-CAM) provided visual interpretability of the models' predictive focus points.

Results

The 3D-EfficientNetB7 model outperformed the other models, achieving an accuracy of 0.91 and an AUC of 0.94 after 20 training epochs. It demonstrated particularly effective in the accurate detection and localization of subtle and complex fractures. Grad-CAM visualizations confirmed the model's focus on clinically relevant areas, aligning with expert assessments and enhancing trust in automated diagnostics. Spatial localization techniques were pivotal in improving interpretability, offering clear visual guidance for pinpointing fracture sites.

Conclusion

Our findings highlight the effectiveness of the 3D-EfficientNetB7 model in diagnosing ankle fractures, supported by robust performance metrics and enhanced visualization tools.

A Survey on Explainable Artificial Intelligence (XAI) Techniques for Visualizing Deep Learning Models in Medical Imaging

The combination of medical imaging and deep learning has significantly improved diagnostic and prognostic capabilities in the healthcare domain. Nevertheless, the inherent complexity of deep learning models poses challenges in understanding their decision-making processes. Interpretability and visualization techniques have emerged as crucial tools to unravel the black-box nature of these models, providing insights into their inner workings and enhancing trust in their predictions. This survey paper comprehensively examines various interpretation and visualization techniques applied to deep learning models in medical imaging. The paper reviews methodologies, discusses their applications, and evaluates their effectiveness in enhancing the interpretability, reliability, and clinical relevance of deep learning models in medical image analysis.

A systematic review of deep learning-based spinal bone lesion detection in medical images

Spinal bone lesions encompass a wide array of pathologies, spanning from benign abnormalities to aggressive malignancies, such as diffusely localized metastases. Early detection and accurate differentiation of the underlying diseases is crucial for every patient's clinical treatment and outcome, with radiological imaging being a core element in the diagnostic pathway. Across numerous pathologies and imaging techniques, deep learning (DL) models are progressively considered a valuable resource in the clinical setting. This review describes not only the diagnostic performance of these models and the differing approaches in the field of spinal bone malignancy recognition, but also the lack of standardized methodology and reporting that we believe is currently hampering this newly founded area of research. In line with their established and reliable role in lesion detection, this publication focuses on both computed tomography and magnetic resonance imaging, as well as various derivative modalities (i.e. SPECT). After conducting a systematic literature search and subsequent analysis for applicability and quality using a modified QUADAS-2 scoring system, we confirmed that most of the 14 identified studies were plagued by major limitations, such as insufficient reporting of model statistics and data acquisition, a lacking external validation dataset, and potentially biased annotation. Although we experienced these limitations, we nonetheless conclude that the potential of these methods shines through in the presented results. These findings underline the need for more stringent quality controls in DL studies, as well as model development to afford increased insight and progress in this promising novel field.

Linked Color Imaging with Artificial Intelligence Improves the Detection of Early Gastric Cancer

INTRODUCTION

Esophagogastroduodenoscopy is the most important tool to detect gastric cancer (GC). In this study, we developed a computer-aided detection (CADe) system to detect GC with white light imaging (WLI) and linked color imaging (LCI) modes and aimed to compare the performance of CADe with that of endoscopists.

METHODS

The system was developed based on the deep learning framework from 9,021 images in 385 patients between 2017 and 2020. A total of 116 LCI and WLI videos from 110 patients between 2017 and 2023 were used to evaluate per-case sensitivity and per-frame specificity.

RESULTS

The per-case sensitivity and per-frame specificity of CADe with a confidence level of 0.5 in detecting GC were 78.6% and 93.4% for WLI and 94.0% and 93.3% for LCI, respectively (p < 0.001). The per-case sensitivities of nonexpert endoscopists for WLI and LCI were 45.8% and 80.4%, whereas those of expert endoscopists were 66.7% and 90.6%, respectively. Regarding detectability between CADe and endoscopists, the per-case sensitivities for WLI and LCI were 78.6% and 94.0% in CADe, respectively, which were significantly higher than those for LCI in experts (90.6%, p = 0.004) and those for WLI and LCI in nonexperts (45.8% and 80.4%, respectively, p < 0.001); however, no significant difference for WLI was observed between CADe and experts (p = 0.134).

CONCLUSIONS

Our CADe system showed significantly better sensitivity in detecting GC when used in LCI compared with WLI mode. Moreover, the sensitivity of CADe using LCI is significantly higher than those of expert endoscopists using LCI to detect GC.

A New Deep Learning Algorithm for Detecting Spinal Metastases on Computed Tomography Images

STUDY DESIGN

Retrospective diagnostic study.

OBJECTIVE

To automatically detect osteolytic bone metastasis lesions in the thoracolumbar region using conventional computed tomography (CT) scans, we developed a new deep learning (DL)-based computer-aided detection model.

SUMMARY OF BACKGROUND DATA

Radiographic detection of bone metastasis is often difficult, even for orthopedic surgeons and diagnostic radiologists, with a consequent risk for pathologic fracture or spinal cord injury. If we can improve detection rates, we will be able to prevent the deterioration of patients' quality of life at the end stage of cancer.

MATERIALS AND METHODS

This study included CT scans acquired at Tokyo Medical and Dental University (TMDU) Hospital between 2016 and 2022. A total of 263 positive CT scans that included at least one osteolytic bone metastasis lesion in the thoracolumbar spine and 172 negative CT scans without bone metastasis were collected for the datasets to train and validate the DL algorithm. As a test data set, 20 positive and 20 negative CT scans were separately collected from the training and validation datasets. To evaluate the performance of the established artificial intelligence (AI) model, sensitivity, precision, F1-score, and specificity were calculated. The clinical utility of our AI model was also evaluated through observer studies involving six orthopaedic surgeons and six radiologists.

RESULTS

Our AI model showed a sensitivity, precision, and F1-score of 0.78, 0.68, and 0.72 (per slice) and 0.75, 0.36, and 0.48 (per lesion), respectively. The observer studies revealed that our AI model had comparable sensitivity to orthopaedic or radiology experts and improved the sensitivity and F1-score of residents.

CONCLUSION

We developed a novel DL-based AI model for detecting osteolytic bone metastases in the thoracolumbar spine. Although further improvement in accuracy is needed, the current AI model may be applied to current clinical practice.

LEVEL OF EVIDENCE

Level III.

An Improved Nested U-Net Network for Fluorescence In Situ Hybridization Cell Image Segmentation

Fluorescence in situ hybridization (FISH) is a powerful cytogenetic method used to precisely detect and localize nucleic acid sequences. This technique is proving to be an invaluable tool in medical diagnostics and has made significant contributions to biology and the life sciences. However, the number of cells is large and the nucleic acid sequences are disorganized in the FISH images taken using the microscope. Processing and analyzing images is a time-consuming and laborious task for researchers, as it can easily tire the human eyes and lead to errors in judgment. In recent years, deep learning has made significant progress in the field of medical imaging, especially the successful application of introducing the attention mechanism. The attention mechanism, as a key component of deep learning, improves the understanding and interpretation of medical images by giving different weights to different regions of the image, enabling the model to focus more on important features. To address the challenges in FISH image analysis, we combined medical imaging with deep learning to develop the SEAM-Unet++ automated cell contour segmentation algorithm with integrated attention mechanism. The significant advantage of this algorithm is that it improves the accuracy of cell contours in FISH images. Experiments have demonstrated that by introducing the attention mechanism, our method is able to segment cells that are adherent to each other more efficiently.

Designing a Hybrid Method of Artificial Neural Network and Particle Swarm Optimization to Diagnosis Polyps from Colorectal CT Images

Background

Since colorectal cancer is one of the most important types of cancer in the world that often leads to death, computer-aided diagnostic (CAD) systems are a promising solution for early diagnosis of this disease with fewer side effects than conventional colonoscopy. Therefore, the aim of this research is to design a CAD system for processing colorectal Computerized Tomography (CT) images using a combination of an artificial neural network and a particle swarm optimizer.

Method

First, the data set of the research was created from the colorectal CT images of the patients of Loghman-e Hakim Hospitals in Tehran and Al-Zahra Hospitals in Isfahan who underwent colorectal CT imaging and had conventional colonoscopy done within a maximum period of one month after that. Then the steps of model implementation, including electronic cleansing of images, segmentation, labeling of samples, extraction of features, and training and optimization of the artificial neural network (ANN) with a particle swarm optimizer, were performed. A binomial statistical test and confusion matrix calculation were used to evaluate the model.

Results

The values of accuracy, sensitivity, and specificity of the model with a P value = 0.000 as a result of the McNemar test were 0.9354, 0.9298, and 0.9889, respectively. Also, the result of the P value of the binomial test of the ratio of diagnosis of the model and the radiologist from Loqman Hakim and Al-Zahra Hospitals was 0.044 and 0.021, respectively.

Conclusions

The results of statistical tests and research variables show the efficiency of the CTC-CAD system created based on the hybrid of the ANN and particle swarm optimization compared to the opinion of radiologists in diagnosing colorectal polyps from CTC images.

Supraspinatus extraction from MRI based on attention-dense spatial pyramid UNet network

BACKGROUND

With potential of deep learning in musculoskeletal image interpretation being explored, this paper focuses on the common site of rotator cuff tears, the supraspinatus. It aims to propose and validate a deep learning model to automatically extract the supraspinatus, verifying its superiority through comparison with several classical image segmentation models.

METHOD

Imaging data were retrospectively collected from 60 patients who underwent inpatient treatment for rotator cuff tears at a hospital between March 2021 and May 2023. A dataset of the supraspinatus from MRI was constructed after collecting, filtering, and manually annotating at the pixel level. This paper proposes a novel A-DAsppUnet network that can automatically extract the supraspinatus after training and optimization. The analysis of model performance is based on three evaluation metrics: precision, intersection over union, and Dice coefficient.

RESULTS

The experimental results demonstrate that the precision, intersection over union, and Dice coefficients of the proposed model are 99.20%, 83.38%, and 90.94%, respectively. Furthermore, the proposed model exhibited significant advantages over the compared models.

CONCLUSION

The designed model in this paper accurately extracts the supraspinatus from MRI, and the extraction results are complete and continuous with clear boundaries. The feasibility of using deep learning methods for musculoskeletal extraction and assisting in clinical decision-making was verified. This research holds practical significance and application value in the field of utilizing artificial intelligence for assisting medical decision-making.

Seeking an optimal approach for Computer-aided Diagnosis of Pulmonary Embolism

Pulmonary Embolism (PE) represents a thrombus ("blood clot"), usually originating from a lower extremity vein, that travels to the blood vessels in the lung, causing vascular obstruction and in some patients death. This disorder is commonly diagnosed using Computed Tomography Pulmonary Angiography (CTPA). Deep learning holds great promise for the Computer-aided Diagnosis (CAD) of PE. However, numerous deep learning methods, such as Convolutional Neural Networks (CNN) and Transformer-based models, exist for a given task, causing great confusion regarding the development of CAD systems for PE. To address this confusion, we present a comprehensive analysis of competing deep learning methods applicable to PE diagnosis based on four datasets. First, we use the RSNA PE dataset, which includes (weak) slice-level and exam-level labels, for PE classification and diagnosis, respectively. At the slice level, we compare CNNs with the Vision Transformer (ViT) and the Swin Transformer. We also investigate the impact of self-supervised versus (fully) supervised ImageNet pre-training, and transfer learning over training models from scratch. Additionally, at the exam level, we compare sequence model learning with our proposed transformer-based architecture, Embedding-based ViT (E-ViT). For the second and third datasets, we utilize the CAD-PE Challenge Dataset and Ferdowsi University of Mashad's PE Dataset, where we convert (strong) clot-level masks into slice-level annotations to evaluate the optimal CNN model for slice-level PE classification. Finally, we use our in-house PE-CAD dataset, which contains (strong) clot-level masks. Here, we investigate the impact of our vessel-oriented image representations and self-supervised pre-training on PE false positive reduction at the clot level across image dimensions (2D, 2.5D, and 3D). Our experiments show that (1) transfer learning boosts performance despite differences between photographic images and CTPA scans; (2) self-supervised pre-training can surpass (fully) supervised pre-training; (3) transformer-based models demonstrate comparable performance but slower convergence compared with CNNs for slice-level PE classification; (4) model trained on the RSNA PE dataset demonstrates promising performance when tested on unseen datasets for slice-level PE classification; (5) our E-ViT framework excels in handling variable numbers of slices and outperforms sequence model learning for exam-level diagnosis; and (6) vessel-oriented image representation and self-supervised pre-training both enhance performance for PE false positive reduction across image dimensions. Our optimal approach surpasses state-of-the-art results on the RSNA PE dataset, enhancing AUC by 0.62% (slice-level) and 2.22% (exam-level). On our in-house PE-CAD dataset, 3D vessel-oriented images improve performance from 80.07% to 91.35%, a remarkable 11% gain. Codes are available at GitHub.com/JLiangLab/CAD_PE.

Multiple semantic X-ray medical image retrieval using efficient feature vector extracted by FPN

OBJECTIVE

Content-based medical image retrieval (CBMIR) has become an important part of computer-aided diagnostics (CAD) systems. The complex medical semantic information inherent in medical images is the most difficult part to improve the accuracy of image retrieval. Highly expressive feature vectors play a crucial role in the search process. In this paper, we propose an effective deep convolutional neural network (CNN) model to extract concise feature vectors for multiple semantic X-ray medical image retrieval.

METHODS

We build a feature pyramid based CNN model with ResNet50V2 backbone to extract multi-level semantic information. And we use the well-known public multiple semantic annotated X-ray medical image data set IRMA to train and test the proposed model.

RESULTS

Our method achieves an IRMA error of 32.2, which is the best score compared to the existing literature on this dataset.

CONCLUSIONS

The proposed CNN model can effectively extract multi-level semantic information from X-ray medical images. The concise feature vectors can improve the retrieval accuracy of multi-semantic and unevenly distributed X-ray medical images.

Deep learning in mesoscale brain image analysis: A review

Mesoscale microscopy images of the brain contain a wealth of information which can help us understand the working mechanisms of the brain. However, it is a challenging task to process and analyze these data because of the large size of the images, their high noise levels, the complex morphology of the brain from the cellular to the regional and anatomical levels, the inhomogeneous distribution of fluorescent labels in the cells and tissues, and imaging artifacts. Due to their impressive ability to extract relevant information from images, deep learning algorithms are widely applied to microscopy images of the brain to address these challenges and they perform superiorly in a wide range of microscopy image processing and analysis tasks. This article reviews the applications of deep learning algorithms in brain mesoscale microscopy image processing and analysis, including image synthesis, image segmentation, object detection, and neuron reconstruction and analysis. We also discuss the difficulties of each task and possible directions for further research.

Multiparametric Oncologic Hybrid Imaging: Machine Learning Challenges and Opportunities

BACKGROUND

Machine learning (ML) is considered an important technology for future data analysis in health care.

METHODS

The inherently technology-driven fields of diagnostic radiology and nuclear medicine will both benefit from ML in terms of image acquisition and reconstruction. Within the next few years, this will lead to accelerated image acquisition, improved image quality, a reduction of motion artifacts and - for PET imaging - reduced radiation exposure and new approaches for attenuation correction. Furthermore, ML has the potential to support decision making by a combined analysis of data derived from different modalities, especially in oncology. In this context, we see great potential for ML in multiparametric hybrid imaging and the development of imaging biomarkers.

RESULTS AND CONCLUSION

In this review, we will describe the basics of ML, present approaches in hybrid imaging of MRI, CT, and PET, and discuss the specific challenges associated with it and the steps ahead to make ML a diagnostic and clinical tool in the future.

KEY POINTS

· ML provides a viable clinical solution for the reconstruction, processing, and analysis of hybrid imaging obtained from MRI, CT, and PET..

Clinical Interpretability of Deep Learning for Predicting Microvascular Invasion in Hepatocellular Carcinoma by Using Attention Mechanism

Preoperative prediction of microvascular invasion (MVI) is essential for management decision in hepatocellular carcinoma (HCC). Deep learning-based prediction models of MVI are numerous but lack clinical interpretation due to their "black-box" nature. Consequently, we aimed to use an attention-guided feature fusion network, including intra- and inter-attention modules, to solve this problem. This retrospective study recruited 210 HCC patients who underwent gadoxetate-enhanced MRI examination before surgery. The MRIs on pre-contrast, arterial, portal, and hepatobiliary phases (hepatobiliary phase: HBP) were used to develop single-phase and multi-phase models. Attention weights provided by attention modules were used to obtain visual explanations of predictive decisions. The four-phase fusion model achieved the highest area under the curve (AUC) of 0.92 (95% CI: 0.84-1.00), and the other models proposed AUCs of 0.75-0.91. Attention heatmaps of collaborative-attention layers revealed that tumor margins in all phases and peritumoral areas in the arterial phase and HBP were salient regions for MVI prediction. Heatmaps of weights in fully connected layers showed that the HBP contributed the most to MVI prediction. Our study firstly implemented self-attention and collaborative-attention to reveal the relationship between deep features and MVI, improving the clinical interpretation of prediction models. The clinical interpretability offers radiologists and clinicians more confidence to apply deep learning models in clinical practice, helping HCC patients formulate personalized therapies.

Integrating plan complexity and dosiomics features with deep learning in patient-specific quality assurance for volumetric modulated arc therapy

PURPOSE

To investigate the feasibility and performance of deep learning (DL) models combined with plan complexity (PC) and dosiomics features in the patient-specific quality assurance (PSQA) for patients underwent volumetric modulated arc therapy (VMAT).

METHODS

Total of 201 VMAT plans with measured PSQA results were retrospectively enrolled and divided into training and testing sets randomly at 7:3. PC metrics were calculated using house-built algorithm based on Matlab. Dosiomics features were extracted and selected using Random Forest (RF) from planning target volume (PTV) and overlap regions with 3D dose distributions. The top 50 dosiomics and 5 PC features were selected based on feature importance screening. A DL DenseNet was adapted and trained for the PSQA prediction.

RESULTS

The measured average gamma passing rate (GPR) of these VMAT plans was 97.94% ± 1.87%, 94.33% ± 3.22%, and 87.27% ± 4.81% at the criteria of 3%/3 mm, 3%/2 mm, and 2%/2 mm, respectively. Models with PC features alone demonstrated the lowest area under curve (AUC). The AUC and sensitivity of PC and dosiomics (D) combined model at 2%/2 mm were 0.915 and 0.833, respectively. The AUCs of DL models were improved from 0.943, 0.849, 0.841 to 0.948, 0.890, 0.942 in the combined models (PC + D + DL) at 3%/3 mm, 3%/2 mm and 2%/2 mm, respectively. A best AUC of 0.942 with a sensitivity, specificity and accuracy of 100%, 81.8%, and 83.6% was achieved with combined model (PC + D + DL) at 2%/2 mm.

CONCLUSIONS

Integrating DL with dosiomics and PC metrics is promising in the prediction of GPRs in PSQA for patients underwent VMAT.

Radiomics and artificial neural networks modelling for identification of high-risk carotid plaques

Objective

In this study, we aimed to investigate the classification of symptomatic plaques by evaluating the models generated via two different approaches, a radiomics-based machine learning (ML) approach, and an end-to-end learning approach which utilized deep learning (DL) techniques with several representative model frameworks.

Methods

We collected high-resolution magnetic resonance imaging (HRMRI) data from 104 patients with carotid artery stenosis, who were diagnosed with either symptomatic plaques (SPs) or asymptomatic plaques (ASPs), in two medical centers. 74 patients were diagnosed with SPs and 30 patients were ASPs. Sampling Perfection with Application-optimized Contrasts (SPACE) by using different flip angle Evolutions was used for MRI imaging. Repeated stratified five-fold cross-validation was used to evaluate the accuracy and receiver operating characteristic (ROC) of the trained classifier. The two proposed approaches were investigated to train the models separately. The difference in the model performance of the two proposed methods was quantitatively evaluated to find a better model to differentiate between SPs and ASPs.

Results

3D-SE-Densenet-121 model showed the best performance among all prediction models (AUC, accuracy, precision, sensitivity, and F1-score of 0.9300, 0.9308, 0.9008, 0.8588, and 0.8614, respectively), which were 0.0689, 0.1119, 0.1043, 0.0805, and 0.1089 higher than the best radiomics-based ML model (MLP). Decision curve analysis showed that the 3D-SE-Densenet-121 model delivered more net benefit than the best radiomics-based ML model (MLP) with a wider threshold probability.

Conclusion

The DL models were able to accurately differentiate between symptomatic and asymptomatic carotid plaques with limited data, which outperformed radiomics-based ML models in identifying symptomatic plaques.

Multi-planar 2.5D U-Net for image quality enhancement of dental cone-beam CT

Cone-beam computed tomography (CBCT) can provide 3D images of a targeted area with the advantage of lower dosage than multidetector computed tomography (MDCT; also simply referred to as CT). However, in CBCT, due to the cone-shaped geometry of the X-ray source and the absence of post-patient collimation, the presence of more scattering rays deteriorates the image quality compared with MDCT. CBCT is commonly used in dental clinics, and image artifacts negatively affect the radiology workflow and diagnosis. Studies have attempted to eliminate image artifacts and improve image quality; however, a vast majority of that work sacrificed structural details of the image. The current study presents a novel approach to reduce image artifacts while preserving details and sharpness in the original CBCT image for precise diagnostic purposes. We used MDCT images as reference high-quality images. Pairs of CBCT and MDCT scans were collected retrospectively at a university hospital, followed by co-registration between the CBCT and MDCT images. A contextual loss-optimized multi-planar 2.5D U-Net was proposed. Images corrected using this model were evaluated quantitatively and qualitatively by dental clinicians. The quantitative metrics showed superior quality in output images compared to the original CBCT. In the qualitative evaluation, the generated images presented significantly higher scores for artifacts, noise, resolution, and overall image quality. This proposed novel approach for noise and artifact reduction with sharpness preservation in CBCT suggests the potential of this method for diagnostic imaging.

Computational Methods Summarizing Mutational Patterns in Cancer: Promise and Limitations for Clinical Applications

Since the rise of next-generation sequencing technologies, the catalogue of mutations in cancer has been continuously expanding. To address the complexity of the cancer-genomic landscape and extract meaningful insights, numerous computational approaches have been developed over the last two decades. In this review, we survey the current leading computational methods to derive intricate mutational patterns in the context of clinical relevance. We begin with mutation signatures, explaining first how mutation signatures were developed and then examining the utility of studies using mutation signatures to correlate environmental effects on the cancer genome. Next, we examine current clinical research that employs mutation signatures and discuss the potential use cases and challenges of mutation signatures in clinical decision-making. We then examine computational studies developing tools to investigate complex patterns of mutations beyond the context of mutational signatures. We survey methods to identify cancer-driver genes, from single-driver studies to pathway and network analyses. In addition, we review methods inferring complex combinations of mutations for clinical tasks and using mutations integrated with multi-omics data to better predict cancer phenotypes. We examine the use of these tools for either discovery or prediction, including prediction of tumor origin, treatment outcomes, prognosis, and cancer typing. We further discuss the main limitations preventing widespread clinical integration of computational tools for the diagnosis and treatment of cancer. We end by proposing solutions to address these challenges using recent advances in machine learning.

Classifying Histopathological Images of Oral Squamous Cell Carcinoma using Deep Transfer Learning

Oral cancer is a prevalent malignancy that affects the oral cavity in the region of head and neck. The study of oral malignant lesions is an essential step for the clinicians to provide a better treatment plan at an early stage for oral cancer. Deep learning based computer-aided diagnostic system has achieved success in many applications and can provide an accurate and timely diagnosis of oral malignant lesions. In biomedical image classification, getting large training dataset is a challenge, which can be efficiently handled by transfer learning as it retrieves the general features from a dataset of natural images and adapted directly to new image dataset. In this work, to achieve an effective deep learning based computer-aided system, the classifications of Oral Squamous Cell Carcinoma (OSCC) histopathology images are performed using two proposed approaches. In the first approach, to identify the best appropriate model to differentiate between benign and malignant cancers, transfer learning assisted deep convolutional neural networks (DCNNs), are considered. To handle the challenge of small dataset and further increase the training efficiency of the proposed model, the pretrained VGG16, VGG19, ResNet50, InceptionV3, and MobileNet, are fine-tuned by training half of the layers and leaving others frozen. In the second approach, a baseline DCNN architecture, trained from scratch with 10 convolution layers is proposed. In addition, a comparative analysis of these models is carried out in terms of classification accuracy and other performance measures. The experimental results demonstrate that ResNet50 obtains substantially superior performance than selected fine-tuned DCNN models as well as the proposed baseline model with an accuracy of 96.6%, precision and recall values are 97% and 96%, respectively.

Use of artificial intelligence for the diagnosis of cholesteatoma

Objectives

Accurate diagnosis of cholesteatomas is crucial. However, cholesteatomas can easily be missed in routine otoscopic exams. Convolutional neural networks (CNNs) have performed well in medical image classification, so we evaluated their use for detecting cholesteatomas in otoscopic images.

Study Design

Design and evaluation of artificial intelligence driven workflow for cholesteatoma diagnosis.

Methods

Otoscopic images collected from the faculty practice of the senior author were deidentified and labeled by the senior author as cholesteatoma, abnormal non-cholesteatoma, or normal. An image classification workflow was developed to automatically differentiate cholesteatomas from other possible tympanic membrane appearances. Eight pretrained CNNs were trained on our otoscopic images, then tested on a withheld subset of images to evaluate their final performance. CNN intermediate activations were also extracted to visualize important image features.

Results

A total of 834 otoscopic images were collected, further categorized into 197 cholesteatoma, 457 abnormal non-cholesteatoma, and 180 normal. Final trained CNNs demonstrated strong performance, achieving accuracies of 83.8%-98.5% for differentiating cholesteatoma from normal, 75.6%-90.1% for differentiating cholesteatoma from abnormal non-cholesteatoma, and 87.0%-90.4% for differentiating cholesteatoma from non-cholesteatoma (abnormal non-cholesteatoma + normal). DenseNet201 (100% sensitivity, 97.1% specificity), NASNetLarge (100% sensitivity, 88.2% specificity), and MobileNetV2 (94.1% sensitivity, 100% specificity) were among the best performing CNNs in distinguishing cholesteatoma versus normal. Visualization of intermediate activations showed robust detection of relevant image features by the CNNs.

Conclusion

While further refinement and more training images are needed to improve performance, artificial intelligence-driven analysis of otoscopic images shows great promise as a diagnostic tool for detecting cholesteatomas.

Level of Evidence

3.

Optimization Algorithm of Moving Object Detection Using Multiscale Pyramid Convolutional Neural Networks

Object detection and recognition is a very important topic with significant research value. This research develops an optimised model of moving target identification based on CNN to address the issues of insufficient positioning information and low target detection accuracy (convolutional neural network). In this article, the target classification information and semantic location information are obtained through the fusion of the target detection model and the depth semantic segmentation model. The classification and position portion of the target detection model is provided by the simultaneous fusion of the image features carrying various information and a pyramid structure of multiscale image features so that the matched image fusion characteristics can be used by the target detection model to detect targets of various sizes and shapes. According to experimental findings, this method's accuracy rate is 0.941, which is 0.189 higher than that of the LSTM-NMS algorithm. Through the migration of CNN and the learning of context information, this technique has great robustness and enhances the scene adaptability of feature extraction as well as the accuracy of moving target position detection.

Real-time artificial intelligence (AI)-aided endoscopy improves adenoma detection rates even in experienced endoscopists: a cohort study in Singapore

BACKGROUND

Colonoscopy is a mainstay to detect premalignant neoplastic lesions in the colon. Real-time Artificial Intelligence (AI)-aided colonoscopy purportedly improves the polyp detection rate, especially for small flat lesions. The aim of this study is to evaluate the performance of real-time AI-aided colonoscopy in the detection of colonic polyps.

METHODS

A prospective single institution cohort study was conducted in Singapore. All real-time AI-aided colonoscopies, regardless of indication, performed by specialist-grade endoscopists were anonymously recorded from July to September 2021 and reviewed by 2 independent authors (FHK, JL). Sustained detection of an area by the program was regarded as a "hit". Histology for the polypectomies were reviewed to determine adenoma detection rate (ADR). Individual endoscopist's performance with AI were compared against their baseline performance without AI endoscopy.

RESULTS

A total of 24 (82.8%) endoscopists participated with 18 (62.1%) performing ≥ 5 AI-aided colonoscopies. Of the 18, 72.2% (n = 13) were general surgeons. During that 3-months period, 487 "hits" encountered in 298 colonoscopies. Polypectomies were performed for 51.3% and 68.4% of these polypectomies were adenomas on histology. The post-intervention median ADR was 30.4% was higher than the median baseline polypectomy rate of 24.3% (p = 0.02). Of the adenomas excised, 14 (5.6%) were sessile serrated adenomas. Of those who performed ≥ 5 AI-aided colonoscopies, 13 (72.2%) had an improvement of ADR compared to their polypectomy rate before the introduction of AI, of which 2 of them had significant improvement.

CONCLUSIONS

Real-time AI-aided colonoscopy have the potential to improved ADR even for experienced endoscopists and would therefore, improve the quality of colonoscopy.

DLNLF-net: Denoised local and non-local deep features fusion network for malignancy characterization of hepatocellular carcinoma

INTRODUCTION

Hepatocellular carcinoma (HCC) is a primary liver cancer with high mortality rate. The degree of HCC malignancy is an important prognostic factor for predicting recurrence and survival after surgical resection or liver transplantation in clinical practice. Currently, deep features obtained from data-driven machine learning algorithms have demonstrated superior performance in characterising lesion features in medical imaging processing. However, previous convolutional neural network (CNN)-based studies on HCC lesion characterisation were based on traditional local deep features. The aim of this study was to propose a denoised local and non-local deep features fusion network (DLNLF-net) for grading HCC.

METHODS

Gadolinium-diethylenetriaminepentaacetic-acid-enhanced magnetic resonance imaging data of 117 histopathologically proven HCCs were collected from 112 patients with resected HCC between October 2012 and October 2018. The proposed DLNLF-net primarily consists of three modules: feature denoising, non-local feature extraction, and bilinear kernel fusion. First, local feature maps were extracted from the original tumour images using convolution operations, followed by a feature denoising block to generate denoised local features. Simultaneously, a non-local feature extraction block was employed on the local feature maps to generate non-local features. Finally, the two generated features were fused using a bilinear kernel model to output the classification results. The dataset was divided into a training set (77 HCC images) and an independent test set (40 HCC images). Training and independent testing were repeated five times to reduce measurement errors. Accuracy, sensitivity, specificity, and area under the curve (AUC) values in the five repetitive tests were calculated to evaluate the performance of the proposed method.

RESULTS

Denoised local features (AUC 89.19%) and non-local features (AUC 88.28%) showed better performance than local features (AUC 86.21%) and global average pooling features (AUC 87.1%) that were derived from a CNN for malignancy characterisation of HCC. Furthermore, the proposed DLNFL-net yielded superior performance (AUC 94.89%) than a typical 3D CNN (AUC 86.21%), bilinear CNN (AUC 90.46%), recently proposed local and global diffusion method (AUC 93.94%), and convolutional block attention module method (AUC 93.62%) for malignancy characterisation of HCC.

CONCLUSION

The non-local operation demonstrated a better capability of yielding global representation, and feature denoising based on the non-local operation achieved performance gains for lesion characterisation. The proposed DLNLF-net, which integrates denoised local and non-local deep features, evidently outperforms conventional CNN-based methods in the malignancy characterisation of HCC.

Towards automated organs at risk and target volumes contouring: Defining precision radiation therapy in the modern era

Precision radiotherapy is a critical and indispensable cancer treatment means in the modern clinical workflow with the goal of achieving "quality-up and cost-down" in patient care. The challenge of this therapy lies in developing computerized clinical-assistant solutions with precision, automation, and reproducibility built-in to deliver it at scale. In this work, we provide a comprehensive yet ongoing, incomplete survey of and discussions on the recent progress of utilizing advanced deep learning, semantic organ parsing, multimodal imaging fusion, neural architecture search and medical image analytical techniques to address four corner-stone problems or sub-problems required by all precision radiotherapy workflows, namely, organs at risk (OARs) segmentation, gross tumor volume (GTV) segmentation, metastasized lymph node (LN) detection, and clinical tumor volume (CTV) segmentation. Without loss of generality, we mainly focus on using esophageal and head-and-neck cancers as examples, but the methods can be extrapolated to other types of cancers. High-precision, automated and highly reproducible OAR/GTV/LN/CTV auto-delineation techniques have demonstrated their effectiveness in reducing the inter-practitioner variabilities and the time cost to permit rapid treatment planning and adaptive replanning for the benefit of patients. Through the presentation of the achievements and limitations of these techniques in this review, we hope to encourage more collective multidisciplinary precision radiotherapy workflows to transpire.

Deep learning-based algorithm improved radiologists' performance in bone metastases detection on CT

OBJECTIVES

To develop and evaluate a deep learning-based algorithm (DLA) for automatic detection of bone metastases on CT.

METHODS

This retrospective study included CT scans acquired at a single institution between 2009 and 2019. Positive scans with bone metastases and negative scans without bone metastasis were collected to train the DLA. Another 50 positive and 50 negative scans were collected separately from the training dataset and were divided into validation and test datasets at a 2:3 ratio. The clinical efficacy of the DLA was evaluated in an observer study with board-certified radiologists. Jackknife alternative free-response receiver operating characteristic analysis was used to evaluate observer performance.

RESULTS

A total of 269 positive scans including 1375 bone metastases and 463 negative scans were collected for the training dataset. The number of lesions identified in the validation and test datasets was 49 and 75, respectively. The DLA achieved a sensitivity of 89.8% (44 of 49) with 0.775 false positives per case for the validation dataset and 82.7% (62 of 75) with 0.617 false positives per case for the test dataset. With the DLA, the overall performance of nine radiologists with reference to the weighted alternative free-response receiver operating characteristic figure of merit improved from 0.746 to 0.899 (p < .001). Furthermore, the mean interpretation time per case decreased from 168 to 85 s (p = .004).

CONCLUSION

With the aid of the algorithm, the overall performance of radiologists in bone metastases detection improved, and the interpretation time decreased at the same time.

KEY POINTS

• A deep learning-based algorithm for automatic detection of bone metastases on CT was developed. • In the observer study, overall performance of radiologists in bone metastases detection improved significantly with the aid of the algorithm. • Radiologists' interpretation time decreased at the same time.

Lymph Node Metastasis Prediction From Whole Slide Images With Transformer-Guided Multiinstance Learning and Knowledge Transfer

The gold standard for diagnosing lymph node metastasis of papillary thyroid carcinoma is to analyze the whole slide histopathological images (WSIs). Due to the large size of WSIs, recent computer-aided diagnosis approaches adopt the multi-instance learning (MIL) strategy and the key part is how to effectively aggregate the information of different instances (patches). In this paper, a novel transformer-guided framework is proposed to predict lymph node metastasis from WSIs, where we incorporate the transformer mechanism to improve the accuracy from three different aspects. First, we propose an effective transformer-based module for discriminative patch feature extraction, including a lightweight feature extractor with a pruned transformer (Tiny-ViT) and a clustering-based instance selection scheme. Next, we propose a new Transformer-MIL module to capture the relationship of different discriminative patches with sparse distribution on WSIs and better nonlinearly aggregate patch-level features into the slide-level prediction. Considering that the slide-level annotation is relatively limited to training a robust Transformer-MIL, we utilize the pathological relationship between the primary tumor and its lymph node metastasis and develop an effective attention-based mutual knowledge distillation (AMKD) paradigm. Experimental results on our collected WSI dataset demonstrate the efficiency of the proposed Transformer-MIL and attention-based knowledge distillation. Our method outperforms the state-of-the-art methods by over 2.72% in AUC (area under the curve).

Integrating features from lymph node stations for metastatic lymph node detection

Metastasis on lymph nodes (LNs), the most common way of spread for primary tumor cells, is a sign of increased mortality. However, metastatic LNs are time-consuming and challenging to detect even for professional radiologists due to their small sizes, high sparsity, and ambiguity in appearance. It is desired to leverage recent development in deep learning to automatically detect metastatic LNs. Besides a two-stage detection network, we here introduce an additional branch to leverage information about LN stations, an important reference for radiologists during metastatic LN diagnosis, as supplementary information for metastatic LN detection. The branch targets to solve a closely related task on the LN station level, i.e., classifying whether an LN station contains metastatic LN or not, so as to learn representations for LN stations. Considering that a metastatic LN station is expected to significantly affect the nearby ones, a GCN-based structure is adopted by the branch to model the relationship among different LN stations. At the classification stage of metastatic LN detection, the above learned LN station features, as well as the features reflecting the distance between the LN candidate and the LN stations, are integrated with the LN features. We validate our method on a dataset containing 114 intravenous contrast-enhanced Computed Tomography (CT) images of oral squamous cell carcinoma (OSCC) patients and show that it outperforms several state-of-the-art methods on the mFROC, maxF1, and AUC scores, respectively.

PPsNet: An improved deep learning model for microsatellite instability high prediction in colorectal cancer from whole slide images

BACKGROUND AND OBJECTIVE

Recent studies have shown that colorectal cancer (CRC) patients with microsatellite instability high (MSI-H) are more likely to benefit from immunotherapy. However, current MSI testing methods are not available for all patients due to the lack of available equipment and trained personnel, as well as the high cost of the assay. Here, we developed an improved deep learning model to predict MSI-H in CRC from whole slide images (WSIs).

METHODS

We established the MSI-H prediction model based on two stages: tumor detection and MSI classification. Previous works applied fine-tuning strategy directly for tumor detection, but ignoring the challenge of vanishing gradient due to the large number of convolutional layers. We added auxiliary classifiers to intermediate layers of pre-trained models to help propagate gradients back through in an effective manner. To predict MSI status, we constructed a pair-wise learning model with a synergic network, named parameter partial sharing network (PPsNet), where partial parameters are shared among two deep convolutional neural networks (DCNNs). The proposed PPsNet contained fewer parameters and reduced the problem of intra-class variation and inter-class similarity. We validated the proposed model on a holdout test set and two external test sets.

RESULTS

144 H&E-stained WSIs from 144 CRC patients (81 cases with MSI-H and 63 cases with MSI-L/MSS) were collected retrospectively from three hospitals. The experimental results indicate that deep supervision based fine-tuning almost outperforms training from scratch and utilizing fine-tuning directly. The proposed PPsNet always achieves better accuracy and area under the receiver operating characteristic curve (AUC) than other solutions with four different neural network architectures on validation. The proposed method finally achieves obvious improvements than other state-of-the-art methods on the validation dataset with an accuracy of 87.28% and AUC of 94.29%.

CONCLUSIONS

The proposed method can obviously increase model performance and our model yields better performance than other methods. Additionally, this work also demonstrates the feasibility of MSI-H prediction using digital pathology images based on deep learning in the Asian population. It is hoped that this model could serve as an auxiliary tool to identify CRC patients with MSI-H more time-saving and efficiently.

Cloud Computing-Based Framework for Breast Tumor Image Classification Using Fusion of AlexNet and GLCM Texture Features with Ensemble Multi-Kernel Support Vector Machine (MK-SVM)

Breast cancer is common among women all over the world. Early identification of breast cancer lowers death rates. However, it is difficult to determine whether these are cancerous or noncancerous lesions due to their inconsistencies in image appearance. Machine learning techniques are widely employed in imaging analysis as a diagnostic method for breast cancer classification. However, patients cannot take advantage of remote areas as these systems are unavailable on clouds. Thus, breast cancer detection for remote patients is indispensable, which can only be possible through cloud computing. The user is allowed to feed images into the cloud system, which is further investigated through the computer aided diagnosis (CAD) system. Such systems could also be used to track patients, older adults, especially with disabilities, particularly in remote areas of developing countries that do not have medical facilities and paramedic staff. In the proposed CAD system, a fusion of AlexNet architecture and GLCM (gray-level cooccurrence matrix) features are used to extract distinguishable texture features from breast tissues. Finally, to attain higher precision, an ensemble of MK-SVM is used. For testing purposes, the proposed model is applied to the MIAS dataset, a commonly used breast image database, and achieved 96.26% accuracy.

Development of a Hybrid-Imaging-Based Prognostic Index for Metastasized-Melanoma Patients in Whole-Body 18F-FDG PET/CT and PET/MRI Data

Besides tremendous treatment success in advanced melanoma patients, the rapid development of oncologic treatment options comes with increasingly high costs and can cause severe life-threatening side effects. For this purpose, predictive baseline biomarkers are becoming increasingly important for risk stratification and personalized treatment planning. Thus, the aim of this pilot study was the development of a prognostic tool for the risk stratification of the treatment response and mortality based on PET/MRI and PET/CT, including a convolutional neural network (CNN) for metastasized-melanoma patients before systemic-treatment initiation. The evaluation was based on 37 patients (19 f, 62 ± 13 y/o) with unresectable metastasized melanomas who underwent whole-body 18F-FDG PET/MRI and PET/CT scans on the same day before the initiation of therapy with checkpoint inhibitors and/or BRAF/MEK inhibitors. The overall survival (OS), therapy response, metastatically involved organs, number of lesions, total lesion glycolysis, total metabolic tumor volume (TMTV), peak standardized uptake value (SULpeak), diameter (Dmlesion) and mean apparent diffusion coefficient (ADCmean) were assessed. For each marker, a Kaplan−Meier analysis and the statistical significance (Wilcoxon test, paired t-test and Bonferroni correction) were assessed. Patients were divided into high- and low-risk groups depending on the OS and treatment response. The CNN segmentation and prediction utilized multimodality imaging data for a complementary in-depth risk analysis per patient. The following parameters correlated with longer OS: a TMTV < 50 mL; no metastases in the brain, bone, liver, spleen or pleura; ≤4 affected organ regions; no metastases; a Dmlesion > 37 mm or SULpeak < 1.3; a range of the ADCmean < 600 mm2/s. However, none of the parameters correlated significantly with the stratification of the patients into the high- or low-risk groups. For the CNN, the sensitivity, specificity, PPV and accuracy were 92%, 96%, 92% and 95%, respectively. Imaging biomarkers such as the metastatic involvement of specific organs, a high tumor burden, the presence of at least one large lesion or a high range of intermetastatic diffusivity were negative predictors for the OS, but the identification of high-risk patients was not feasible with the handcrafted parameters. In contrast, the proposed CNN supplied risk stratification with high specificity and sensitivity.

Automated Segmentation of Spinal Muscles From Upright Open MRI Using a Multiscale Pyramid 2D Convolutional Neural Network

STUDY DESIGN

Randomized trial.

OBJECTIVE

To implement an algorithm enabling the automated segmentation of spinal muscles from open magnetic resonance images in healthy volunteers and patients with adult spinal deformity (ASD).

SUMMARY OF BACKGROUND DATA

Understanding spinal muscle anatomy is critical to diagnosing and treating spinal deformity.Muscle boundaries can be extrapolated from medical images using segmentation, which is usually done manually by clinical experts and remains complicated and time-consuming.

METHODS

Three groups were examined: two healthy volunteer groups (N = 6 for each group) and one ASD group (N = 8 patients) were imaged at the lumbar and thoracic regions of the spine in an upright open magnetic resonance imaging scanner while maintaining different postures (various seated, standing, and supine). For each group and region, a selection of regions of interest (ROIs) was manually segmented. A multiscale pyramid two-dimensional convolutional neural network was implemented to automatically segment all defined ROIs. A five-fold crossvalidation method was applied and distinct models were trained for each resulting set and group and evaluated using Dice coefficients calculated between the model output and the manually segmented target.

RESULTS

Good to excellent results were found across all ROIs for the ASD (Dice coefficient >0.76) and healthy (dice coefficient > 0.86) groups.

CONCLUSION

This study represents a fundamental step toward the development of an automated spinal muscle properties extraction pipeline, which will ultimately allow clinicians to have easier access to patient-specific simulations, diagnosis, and treatment.

Deep Learning Mechanism for Predicting the Axillary Lymph Node Metastasis in Patients with Primary Breast Cancer

The second largest cause of mortality worldwide is breast cancer, and it mostly occurs in women. Early diagnosis has improved further treatments and reduced the level of mortality. A unique deep learning algorithm is presented for predicting breast cancer in its early stages. This method utilizes numerous layers to retrieve significantly greater amounts of information from the source inputs. It could perform automatic quantitative evaluation of complicated image properties in the medical field and give greater precision and reliability during the diagnosis. The dataset of axillary lymph nodes from the breast cancer patients was collected from Erasmus Medical Center. A total of 1050 images were studied from the 850 patients during the years 2018 to 2021. For the independent test, data samples were collected for 100 images from 95 patients at national cancer institute. The existence of axillary lymph nodes was confirmed by pathologic examination. The feed forward, radial basis function, and Kohonen self-organizing are the artificial neural networks (ANNs) which are used to train 84% of the Erasmus Medical Center dataset and test the remaining 16% of the independent dataset. The proposed model performance was determined in terms of accuracy (Ac), sensitivity (Sn), specificity (Sf), and the outcome of the receiver operating curve (Roc), which was compared to the other four radiologists' mechanism. The result of the study shows that the proposed mechanism achieves 95% sensitivity, 96% specificity, and 98% accuracy, which is higher than the radiologists' models (90% sensitivity, 92% specificity, and 94% accuracy). Deep learning algorithms could accurately predict the clinical negativity of axillary lymph node metastases by utilizing images of initial breast cancer patients. This method provides an earlier diagnostic technique for axillary lymph node metastases in patients with medically negative changes in axillary lymph nodes.

CAN3D: Fast 3D medical image segmentation via compact context aggregation

Direct automatic segmentation of objects in 3D medical imaging, such as magnetic resonance (MR) imaging, is challenging as it often involves accurately identifying multiple individual structures with complex geometries within a large volume under investigation. Most deep learning approaches address these challenges by enhancing their learning capability through a substantial increase in trainable parameters within their models. An increased model complexity will incur high computational costs and large memory requirements unsuitable for real-time implementation on standard clinical workstations, as clinical imaging systems typically have low-end computer hardware with limited memory and CPU resources only. This paper presents a compact convolutional neural network (CAN3D) designed specifically for clinical workstations and allows the segmentation of large 3D Magnetic Resonance (MR) images in real-time. The proposed CAN3D has a shallow memory footprint to reduce the number of model parameters and computer memory required for state-of-the-art performance and maintain data integrity by directly processing large full-size 3D image input volumes with no patches required. The proposed architecture significantly reduces computational costs, especially for inference using the CPU. We also develop a novel loss function with extra shape constraints to improve segmentation accuracy for imbalanced classes in 3D MR images. Compared to state-of-the-art approaches (U-Net3D, improved U-Net3D and V-Net), CAN3D reduced the number of parameters up to two orders of magnitude and achieved much faster inference, up to 5 times when predicting with a standard commercial CPU (instead of GPU). For the open-access OAI-ZIB knee MR dataset, in comparison with manual segmentation, CAN3D achieved Dice coefficient values of (mean = 0.87 ± 0.02 and 0.85 ± 0.04) with mean surface distance errors (mean = 0.36 ± 0.32 mm and 0.29 ± 0.10 mm) for imbalanced classes such as (femoral and tibial) cartilage volumes respectively when training volume-wise under only 12G video memory. Similarly, CAN3D demonstrated high accuracy and efficiency on a pelvis 3D MR imaging dataset for prostate cancer consisting of 211 examinations with expert manual semantic labels (bladder, body, bone, rectum, prostate) now released publicly for scientific use as part of this work.

An enhanced multi-modal brain graph network for classifying neuropsychiatric disorders

It has been proven that neuropsychiatric disorders (NDs) can be associated with both structures and functions of brain regions. Thus, data about structures and functions could be usefully combined in a comprehensive analysis. While brain structural MRI (sMRI) images contain anatomic and morphological information about NDs, functional MRI (fMRI) images carry complementary information. However, efficient extraction and fusion of sMRI and fMRI data remains challenging. In this study, we develop an enhanced multi-modal graph convolutional network (MME-GCN) in a binary classification between patients with NDs and healthy controls, based on the fusion of the structural and functional graphs of the brain region. First, based on the same brain atlas, we construct structural and functional graphs from sMRI and fMRI data, respectively. Second, we use machine learning to extract important features from the structural graph network. Third, we use these extracted features to adjust the corresponding edge weights in the functional graph network. Finally, we train a multi-layer GCN and use it in binary classification task. MME-GCN achieved 93.71% classification accuracy on the open data set provided by the Consortium for Neuropsychiatric Phenomics. In addition, we analyzed the important features selected from the structural graph and verified them in the functional graph. Using MME-GCN, we found several specific brain connections important to NDs.

Predicting brain structural network using functional connectivity

Uncovering the non-trivial brain structure-function relationship is fundamentally important for revealing organizational principles of human brain. However, it is challenging to infer a reliable relationship between individual brain structure and function, e.g., the relations between individual brain structural connectivity (SC) and functional connectivity (FC). Brain structure-function displays a distributed and heterogeneous pattern, that is, many functional relationships arise from non-overlapping sets of anatomical connections. This complex relation can be interwoven with widely existed individual structural and functional variations. Motivated by the advances of generative adversarial network (GAN) and graph convolutional network (GCN) in the deep learning field, in this work, we proposed a multi-GCN based GAN (MGCN-GAN) to infer individual SC based on corresponding FC by automatically learning the complex associations between individual brain structural and functional networks. The generator of MGCN-GAN is composed of multiple multi-layer GCNs which are designed to model complex indirect connections in brain network. The discriminator of MGCN-GAN is a single multi-layer GCN which aims to distinguish the predicted SC from real SC. To overcome the inherent unstable behavior of GAN, we designed a new structure-preserving (SP) loss function to guide the generator to learn the intrinsic SC patterns more effectively. Using Human Connectome Project (HCP) dataset and Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset as test beds, our MGCN-GAN model can generate reliable individual SC from FC. This result implies that there may exist a common regulation between specific brain structural and functional architectures across different individuals.

Recent advances and clinical applications of deep learning in medical image analysis

Deep learning has received extensive research interest in developing new medical image processing algorithms, and deep learning based models have been remarkably successful in a variety of medical imaging tasks to support disease detection and diagnosis. Despite the success, the further improvement of deep learning models in medical image analysis is majorly bottlenecked by the lack of large-sized and well-annotated datasets. In the past five years, many studies have focused on addressing this challenge. In this paper, we reviewed and summarized these recent studies to provide a comprehensive overview of applying deep learning methods in various medical image analysis tasks. Especially, we emphasize the latest progress and contributions of state-of-the-art unsupervised and semi-supervised deep learning in medical image analysis, which are summarized based on different application scenarios, including classification, segmentation, detection, and image registration. We also discuss major technical challenges and suggest possible solutions in the future research efforts.

Small Object Detection via Pixel Level Balancing With Applications to Blood Cell Detection

Object detection technology has been widely used in medical field, such as detecting the images of blood cell to count the changes and distribution for assisting the diagnosis of diseases. However, detecting small objects is one of the most challenging and important problems especially in medical scenarios. Most of the objects in medical images are very small but influential. Improving the detection performance of small objects is a very meaningful topic for medical detection. Current researches mainly focus on the extraction of small object features and data augmentation for small object samples, all of these researches focus on extracting the feature space of small objects better. However, in the training process of a detection model, objects of different sizes are mixed together, which may interfere with each other and affect the performance of small object detection. In this paper, we propose a method called pixel level balancing (PLB), which takes into account the number of pixels contained in the detection box as an impact factor to characterize the size of the inspected objects, and uses this as an impact factor. The training loss of each object of different size is adjusted by a weight dynamically, so as to improve the accuracy of small object detection. Finally, through experiments, we demonstrate that the size of objects in object detection interfere with each other. So that we can improve the accuracy of small object detection through PLB operation. This method can perform well with blood cell detection in our experiments.

Transfer learning model for false positive reduction in lymph node detection via sparse coding and deep learning

Transfer learning technique is popularly employed for a lot of medical image classification tasks. Here based on convolutional neural network (CNN) and sparse coding process, we present a new deep transfer learning architecture for false positive reduction in lymph node detection task. We first convert the linear combination of the deep transferred features to the pre-trained filter banks. Next, a new point-wise filter based CNN branch is introduced to automatically integrate transfer features for the false and positive image classification purpose. To lower the scale of the proposed architecture, we bring sparse coding process to the fixed transferred convolution filter banks. On this basis, a two-stage training strategy with grouped sparse connection is presented to train the model efficiently. The model validity is tested on lymph node dataset for false positive reduction and our approach indicates encouraging performances compared to prior approaches. Our method reaches sensitivities of 71% /85% at 3 FP/vol. and 82% /91% at 6 FP/vol. in abdomen and mediastinum respectively, which compare competitively to previous approaches.

Toward an artificial intelligence-assisted framework for reconstructing the digital twin of vertebra and predicting its fracture response

This article presents an effort toward building an artificial intelligence (AI) assisted framework, coined ReconGAN, for creating a realistic digital twin of the human vertebra and predicting the risk of vertebral fracture (VF). ReconGAN consists of a deep convolutional generative adversarial network (DCGAN), image-processing steps, and finite element (FE) based shape optimization to reconstruct the vertebra model. This DCGAN model is trained using a set of quantitative micro-computed tomography (micro-QCT) images of the trabecular bone obtained from cadaveric samples. The quality of synthetic trabecular models generated using DCGAN are verified by comparing a set of its statistical microstructural descriptors with those of the imaging data. The synthesized trabecular microstructure is then infused into the vertebra cortical shell extracted from the patient's diagnostic CT scans using an FE-based shape optimization approach to achieve a smooth transition between trabecular to cortical regions. The final geometrical model of the vertebra is converted into a high-fidelity FE model to simulate the VF response using a continuum damage model under compression and flexion loading conditions. A feasibility study is presented to demonstrate the applicability of digital twins generated using this AI-assisted framework to predict the risk of VF in a cancer patient with spinal metastasis.

New open-source software for subcellular segmentation and analysis of spatiotemporal fluorescence signals using deep learning

Cellular imaging instrumentation advancements as well as readily available optogenetic and fluorescence sensors have yielded a profound need for fast, accurate, and standardized analysis. Deep-learning architectures have revolutionized the field of biomedical image analysis and have achieved state-of-the-art accuracy. Despite these advancements, deep learning architectures for the segmentation of subcellular fluorescence signals is lacking. Cellular dynamic fluorescence signals can be plotted and visualized using spatiotemporal maps (STMaps), and currently their segmentation and quantification are hindered by slow workflow speed and lack of accuracy, especially for large datasets. In this study, we provide a software tool that utilizes a deep-learning methodology to fundamentally overcome signal segmentation challenges. The software framework demonstrates highly optimized and accurate calcium signal segmentation and provides a fast analysis pipeline that can accommodate different patterns of signals across multiple cell types. The software allows seamless data accessibility, quantification, and graphical visualization and enables large dataset analysis throughput.

Deep Neural Networks and Machine Learning Radiomics Modelling for Prediction of Relapse in Mantle Cell Lymphoma

Simple Summary

Mantle cell lymphoma (MCL) is an aggressive lymphoid tumour with a poor prognosis. There exist no routine biomarkers for the early prediction of relapse. Our study compared the potential of radiomics-based machine learning and 3D deep learning models as non-invasive biomarkers to risk-stratify MCL patients, thus promoting precision imaging in clinical oncology.

Abstract

Mantle cell lymphoma (MCL) is a rare lymphoid malignancy with a poor prognosis characterised by frequent relapse and short durations of treatment response. Most patients present with aggressive disease, but there exist indolent subtypes without the need for immediate intervention. The very heterogeneous behaviour of MCL is genetically characterised by the translocation t(11;14)(q13;q32), leading to Cyclin D1 overexpression with distinct clinical and biological characteristics and outcomes. There is still an unfulfilled need for precise MCL prognostication in real-time. Machine learning and deep learning neural networks are rapidly advancing technologies with promising results in numerous fields of application. This study develops and compares the performance of deep learning (DL) algorithms and radiomics-based machine learning (ML) models to predict MCL relapse on baseline CT scans. Five classification algorithms were used, including three deep learning models (3D SEResNet50, 3D DenseNet, and an optimised 3D CNN) and two machine learning models based on K-nearest Neighbor (KNN) and Random Forest (RF). The best performing method, our optimised 3D CNN, predicted MCL relapse with a 70% accuracy, better than the 3D SEResNet50 (62%) and the 3D DenseNet (59%). The second-best performing method was the KNN-based machine learning model (64%) after principal component analysis for improved accuracy. Our optimised CNN developed by ourselves correctly predicted MCL relapse in 70% of the patients on baseline CT imaging. Once prospectively tested in clinical trials with a larger sample size, our proposed 3D deep learning model could facilitate clinical management by precision imaging in MCL.

Machine Learning for Early Parkinson’s Disease Identification within SWEDD Group Using Clinical and DaTSCAN SPECT Imaging Features

Early Parkinson’s Disease (PD) diagnosis is a critical challenge in the treatment process. Meeting this challenge allows appropriate planning for patients. However, Scan Without Evidence of Dopaminergic Deficit (SWEDD) is a heterogeneous group of PD patients and Healthy Controls (HC) in clinical and imaging features. The application of diagnostic tools based on Machine Learning (ML) comes into play here as they are capable of distinguishing between HC subjects and PD patients within an SWEDD group. In the present study, three ML algorithms were used to separate PD patients from HC within an SWEDD group. Data of 548 subjects were firstly analyzed by Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) techniques. Using the best reduction technique result, we built the following clustering models: Density-Based Spatial (DBSCAN), K-means and Hierarchical Clustering. According to our findings, LDA performs better than PCA; therefore, LDA was used as input for the clustering models. The different models’ performances were assessed by comparing the clustering algorithms outcomes with the ground truth after a follow-up. Hierarchical Clustering surpassed DBSCAN and K-means algorithms by 64%, 78.13% and 38.89% in terms of accuracy, sensitivity and specificity. The proposed method demonstrated the suitability of ML models to distinguish PD patients from HC subjects within an SWEDD group.