Unpaired deep learning for pharmacokinetic parameter estimation from dynamic contrast-enhanced MRI without AIF measurements

DCE-MRI provides information about vascular permeability and tissue perfusion through the acquisition of pharmacokinetic parameters. However, traditional methods for estimating these pharmacokinetic parameters involve fitting tracer kinetic models, which often suffer from computational complexity and low accuracy due to noisy arterial input function (AIF) measurements. Although some deep learning approaches have been proposed to tackle these challenges, most existing methods rely on supervised learning that requires paired input DCE-MRI and labeled pharmacokinetic parameter maps. This dependency on labeled data introduces significant time and resource constraints and potential noise in the labels, making supervised learning methods often impractical. To address these limitations, we present a novel unpaired deep learning method for estimating pharmacokinetic parameters and the AIF using a physics-driven CycleGAN approach. Our proposed CycleGAN framework is designed based on the underlying physics model, resulting in a simpler architecture with a single generator and discriminator pair. Crucially, our experimental results indicate that our method does not necessitate separate AIF measurements and produces more reliable pharmacokinetic parameters than other techniques.

Coronary plaque phenotype associated with positive remodeling

BACKGROUND

Positive remodeling is an integral part of the vascular adaptation process during the development of atherosclerosis, which can be detected by coronary computed tomography angiography (CTA).

METHODS

A total of 426 patients who underwent both coronary CTA and optical coherence tomography (OCT) were included. Four machine learning (ML) models, gradient boosting machine (GBM), random forest (RF), deep learning (DL), and support vector machine (SVM), were employed to detect specific plaque features. A total of 15 plaque features assessed by OCT were analyzed. The variable importance ranking was used to identify the features most closely associated with positive remodeling.

RESULTS

In the variable importance ranking, lipid index and maximal calcification arc were consistently ranked high across all four ML models. Lipid index and maximal calcification arc were correlated with positive remodeling, showing pronounced influence at the lower range and diminishing influence at the higher range. Patients with more plaques with positive remodeling throughout their entire coronary trees had higher low-density lipoprotein cholesterol levels and were associated with a higher incidence of cardiovascular events during 5-year follow-up (Hazard ratio 2.10 [1.26-3.48], P ​= ​0.004).

CONCLUSION

Greater lipid accumulation and less calcium burden were important features associated with positive remodeling in the coronary arteries. The number of coronary plaques with positive remodeling was associated with a higher incidence of cardiovascular events.

Bidirectional generation of structure and properties through a single molecular foundation model

Recent successes of foundation models in artificial intelligence have prompted the emergence of large-scale chemical pre-trained models. Despite the growing interest in large molecular pre-trained models that provide informative representations for downstream tasks, attempts for multimodal pre-training approaches on the molecule domain were limited. To address this, here we present a multimodal molecular pre-trained model that incorporates the modalities of structure and biochemical properties, drawing inspiration from recent advances in multimodal learning techniques. Our proposed model pipeline of data handling and training objectives aligns the structure/property features in a common embedding space, which enables the model to regard bidirectional information between the molecules' structure and properties. These contributions emerge synergistic knowledge, allowing us to tackle both multimodal and unimodal downstream tasks through a single model. Through extensive experiments, we demonstrate that our model has the capabilities to solve various meaningful chemical challenges, including conditional molecule generation, property prediction, molecule classification, and reaction prediction.

Improving Medical Speech-to-Text Accuracy using Vision-Language Pre-training Models

Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.

Self-supervised multi-modal training from uncurated images and reports enables monitoring AI in radiology

The escalating demand for artificial intelligence (AI) systems that can monitor and supervise human errors and abnormalities in healthcare presents unique challenges. Recent advances in vision-language models reveal the challenges of monitoring AI by understanding both visual and textual concepts and their semantic correspondences. However, there has been limited success in the application of vision-language models in the medical domain. Current vision-language models and learning strategies for photographic images and captions call for a web-scale data corpus of image and text pairs which is not often feasible in the medical domain. To address this, we present a model named medical cross-attention vision-language model (Medical X-VL), which leverages key components to be tailored for the medical domain. The model is based on the following components: self-supervised unimodal models in medical domain and a fusion encoder to bridge them, momentum distillation, sentencewise contrastive learning for medical reports, and sentence similarity-adjusted hard negative mining. We experimentally demonstrated that our model enables various zero-shot tasks for monitoring AI, ranging from the zero-shot classification to zero-shot error correction. Our model outperformed current state-of-the-art models in two medical image datasets, suggesting a novel clinical application of our monitoring AI model to alleviate human errors. Our method demonstrates a more specialized capacity for fine-grained understanding, which presents a distinct advantage particularly applicable to the medical domain.

C-DARL: Contrastive diffusion adversarial representation learning for label-free blood vessel segmentation

Blood vessel segmentation in medical imaging is one of the essential steps for vascular disease diagnosis and interventional planning in a broad spectrum of clinical scenarios in image-based medicine and interventional medicine. Unfortunately, manual annotation of the vessel masks is challenging and resource-intensive due to subtle branches and complex structures. To overcome this issue, this paper presents a self-supervised vessel segmentation method, dubbed the contrastive diffusion adversarial representation learning (C-DARL) model. Our model is composed of a diffusion module and a generation module that learns the distribution of multi-domain blood vessel data by generating synthetic vessel images from diffusion latent. Moreover, we employ contrastive learning through a mask-based contrastive loss so that the model can learn more realistic vessel representations. To validate the efficacy, C-DARL is trained using various vessel datasets, including coronary angiograms, abdominal digital subtraction angiograms, and retinal imaging. Experimental results confirm that our model achieves performance improvement over baseline methods with noise robustness, suggesting the effectiveness of C-DARL for vessel segmentation.Our source code is available at https://github.com/boahK/MEDIA_CDARL.2.

A novel deep learning model for a computed tomography diagnosis of coronary plaque erosion

Patients with acute coronary syndromes caused by plaque erosion might be managed conservatively without stenting. Currently, the diagnosis of plaque erosion requires an invasive imaging procedure. We sought to develop a deep learning (DL) model that enables an accurate diagnosis of plaque erosion using coronary computed tomography angiography (CTA). A total of 532 CTA scans from 395 patients were used to develop a DL model: 426 CTA scans from 316 patients for training and internal validation, and 106 separate scans from 79 patients for validation. Momentum Distillation-enhanced Composite Transformer Attention (MD-CTA), a novel DL model that can effectively process the entire set of CTA scans to diagnose plaque erosion, was developed. The novel DL model, compared to the convolution neural network, showed significantly improved AUC (0.899 [0.841-0.957] vs. 0.724 [0.622-0.826]), sensitivity (87.1 [70.2-96.4] vs. 71.0 [52.0-85.8]), and specificity (85.3 [75.3-92.4] vs. 68.0 [56.2-78.3]), respectively, for the patient-level prediction. Similar results were obtained at the slice-level prediction AUC (0.897 [0.890-0.904] vs. 0.757 [0.744-0.770]), sensitivity (82.2 [79.8-84.3] vs. 68.9 [66.2-71.6]), and specificity (80.1 [79.1-81.0] vs. 67.3 [66.3-68.4]), respectively. This newly developed DL model enables an accurate CT diagnosis of plaque erosion, which might enable cardiologists to provide tailored therapy without invasive procedures.Clinical Trial Registration: http://www.clinicaltrials.gov , NCT04523194.

One-Shot Adaptation of GAN in Just One CLIP

There are many recent research efforts to fine-tune a pre-trained generator with a few target images to generate images of a novel domain. Unfortunately, these methods often suffer from overfitting or under-fitting when fine-tuned with a single target image. To address this, here we present a novel single-shot GAN adaptation method through unified CLIP space manipulations. Specifically, our model employs a two-step training strategy: reference image search in the source generator using a CLIP-guided latent optimization, followed by generator fine-tuning with a novel loss function that imposes CLIP space consistency between the source and adapted generators. To further improve the adapted model to produce spatially consistent samples with respect to the source generator, we also propose contrastive regularization for patchwise relationships in the CLIP space. Experimental results show that our model generates diverse outputs with the target texture and outperforms the baseline models both qualitatively and quantitatively. Furthermore, we show that our CLIP space manipulation strategy allows more effective attribute editing.

Multi-Scale Hybrid Vision Transformer for Learning Gastric Histology: AI-Based Decision Support System for Gastric Cancer Treatment

Gastric endoscopic screening is an effective way to decide appropriate gastric cancer treatment at an early stage, reducing gastric cancer-associated mortality rate. Although artificial intelligence has brought a great promise to assist pathologist to screen digitalized endoscopic biopsies, existing artificial intelligence systems are limited to be utilized in planning gastric cancer treatment. We propose a practical artificial intelligence-based decision support system that enables five subclassifications of gastric cancer pathology, which can be directly matched to general gastric cancer treatment guidance. The proposed framework is designed to efficiently differentiate multi-classes of gastric cancer through multiscale self-attention mechanism using 2-stage hybrid vision transformer networks, by mimicking the way how human pathologists understand histology. The proposed system demonstrates its reliable diagnostic performance by achieving class-average sensitivity of above 0.85 for multicentric cohort tests. Moreover, the proposed system demonstrates its great generalization capability on gastrointestinal track organ cancer by achieving the best class-average sensitivity among contemporary networks. Furthermore, in the observational study, artificial intelligence-assisted pathologists show significantly improved diagnostic sensitivity within saved screening time compared to human pathologists. Our results demonstrate that the proposed artificial intelligence system has a great potential for providing presumptive pathologic opinion and supporting decision of appropriate gastric cancer treatment in practical clinical settings.

Generative Adversarial Network-Based Image Conversion Among Different Computed Tomography Protocols and Vendors: Effects on Accuracy and Variability in Quantifying Regional Disease Patterns of Interstitial Lung Disease

OBJECTIVE

To assess whether computed tomography (CT) conversion across different scan parameters and manufacturers using a routable generative adversarial network (RouteGAN) can improve the accuracy and variability in quantifying interstitial lung disease (ILD) using a deep learning-based automated software.

MATERIALS AND METHODS

This study included patients with ILD who underwent thin-section CT. Unmatched CT images obtained using scanners from four manufacturers (vendors A-D), standard- or low-radiation doses, and sharp or medium kernels were classified into groups 1-7 according to acquisition conditions. CT images in groups 2-7 were converted into the target CT style (Group 1: vendor A, standard dose, and sharp kernel) using a RouteGAN. ILD was quantified on original and converted CT images using a deep learning-based software (Aview, Coreline Soft). The accuracy of quantification was analyzed using the dice similarity coefficient (DSC) and pixel-wise overlap accuracy metrics against manual quantification by a radiologist. Five radiologists evaluated quantification accuracy using a 10-point visual scoring system.

RESULTS

Three hundred and fifty CT slices from 150 patients (mean age: 67.6 ± 10.7 years; 56 females) were included. The overlap accuracies for quantifying total abnormalities in groups 2-7 improved after CT conversion (original vs. converted: 0.63 vs. 0.68 for DSC, 0.66 vs. 0.70 for pixel-wise recall, and 0.68 vs. 0.73 for pixel-wise precision; P < 0.002 for all). The DSCs of fibrosis score, honeycombing, and reticulation significantly increased after CT conversion (0.32 vs. 0.64, 0.19 vs. 0.47, and 0.23 vs. 0.54, P < 0.002 for all), whereas those of ground-glass opacity, consolidation, and emphysema did not change significantly or decreased slightly. The radiologists' scores were significantly higher (P < 0.001) and less variable on converted CT.

CONCLUSION

CT conversion using a RouteGAN can improve the accuracy and variability of CT images obtained using different scan parameters and manufacturers in deep learning-based quantification of ILD.

C-DARL: Contrastive diffusion adversarial representation learning for label-free blood vessel segmentation

Blood vessel segmentation in medical imaging is one of the essential steps for vascular disease diagnosis and interventional planning in a broad spectrum of clinical scenarios in image-based medicine and interventional medicine. Unfortunately, manual annotation of the vessel masks is challenging and resource-intensive due to subtle branches and complex structures. To overcome this issue, this paper presents a self-supervised vessel segmentation method, dubbed the contrastive diffusion adversarial representation learning (C-DARL) model. Our model is composed of a diffusion module and a generation module that learns the distribution of multi-domain blood vessel data by generating synthetic vessel images from diffusion latent. Moreover, we employ contrastive learning through a mask-based contrastive loss so that the model can learn more realistic vessel representations. To validate the efficacy, C-DARL is trained using various vessel datasets, including coronary angiograms, abdominal digital subtraction angiograms, and retinal imaging. Experimental results confirm that our model achieves performance improvement over baseline methods with noise robustness, suggesting the effectiveness of C-DARL for vessel segmentation.

Deep Learning-Enabled Detection of Pneumoperitoneum in Supine and Erect Abdominal Radiography: Modeling Using Transfer Learning and Semi-Supervised Learning

OBJECTIVE

Detection of pneumoperitoneum using abdominal radiography, particularly in the supine position, is often challenging. This study aimed to develop and externally validate a deep learning model for the detection of pneumoperitoneum using supine and erect abdominal radiography.

MATERIALS AND METHODS

A model that can utilize "pneumoperitoneum" and "non-pneumoperitoneum" classes was developed through knowledge distillation. To train the proposed model with limited training data and weak labels, it was trained using a recently proposed semi-supervised learning method called distillation for self-supervised and self-train learning (DISTL), which leverages the Vision Transformer. The proposed model was first pre-trained with chest radiographs to utilize common knowledge between modalities, fine-tuned, and self-trained on labeled and unlabeled abdominal radiographs. The proposed model was trained using data from supine and erect abdominal radiographs. In total, 191212 chest radiographs (CheXpert data) were used for pre-training, and 5518 labeled and 16671 unlabeled abdominal radiographs were used for fine-tuning and self-supervised learning, respectively. The proposed model was internally validated on 389 abdominal radiographs and externally validated on 475 and 798 abdominal radiographs from the two institutions. We evaluated the performance in diagnosing pneumoperitoneum using the area under the receiver operating characteristic curve (AUC) and compared it with that of radiologists.

RESULTS

In the internal validation, the proposed model had an AUC, sensitivity, and specificity of 0.881, 85.4%, and 73.3% and 0.968, 91.1, and 95.0 for supine and erect positions, respectively. In the external validation at the two institutions, the AUCs were 0.835 and 0.852 for the supine position and 0.909 and 0.944 for the erect position. In the reader study, the readers' performances improved with the assistance of the proposed model.

CONCLUSION

The proposed model trained with the DISTL method can accurately detect pneumoperitoneum on abdominal radiography in both the supine and erect positions.

MR Image Denoising and Super-Resolution Using Regularized Reverse Diffusion

Patient scans from MRI often suffer from noise, which hampers the diagnostic capability of such images. As a method to mitigate such artifacts, denoising is largely studied both within the medical imaging community and beyond the community as a general subject. However, recent deep neural network-based approaches mostly rely on the minimum mean squared error (MMSE) estimates, which tend to produce a blurred output. Moreover, such models suffer when deployed in real-world situations: out-of-distribution data, and complex noise distributions that deviate from the usual parametric noise models. In this work, we propose a new denoising method based on score-based reverse diffusion sampling, which overcomes all the aforementioned drawbacks. Our network, trained only with coronal knee scans, excels even on out-of-distribution in vivo liver MRI data, contaminated with a complex mixture of noise. Even more, we propose a method to enhance the resolution of the denoised image with the same network. With extensive experiments, we show that our method establishes state-of-the-art performance while having desirable properties which prior MMSE denoisers did not have: flexibly choosing the extent of denoising, and quantifying uncertainty.

Diagnosis of coronary layered plaque by deep learning

Healed coronary plaques, morphologically characterized by a layered phenotype, are signs of previous plaque destabilization and healing. Recent optical coherence tomography (OCT) studies demonstrated that layered plaque is associated with higher levels of local and systemic inflammation and rapid plaque progression. However, the diagnosis of layered plaque needs expertise in OCT image analysis and is susceptible to inter-observer variability. We developed a deep learning (DL) model for an accurate diagnosis of layered plaque. A Visual Transformer (ViT)-based DL model that integrates information from adjacent frames emulating the cardiologists who review consecutive OCT frames to make a diagnosis was developed and compared with the standard convolutional neural network (CNN) model. A total of 237,021 cross-sectional OCT images from 581 patients collected from 8 sites were used for training and internal validation, and 65,394 images from 292 patients collected from another site were used for external validation. In the five-fold cross-validation, the ViT-based model provided better performance (area under the curve [AUC]: 0.860; 95% confidence interval [CI]: 0.855-0.866) than the standard CNN-based model (AUC: 0.799; 95% CI: 0.792-0.805). The ViT-based model (AUC: 0.845; 95% CI: 0.837-0.853) also surpassed the standard CNN-based model (AUC: 0.791; 95% CI: 0.782-0.800) in the external validation. The ViT-based DL model can accurately diagnose a layered plaque, which could help risk stratification for cardiac events.

Task-Agnostic Vision Transformer for Distributed Learning of Image Processing

Recently, distributed learning approaches have been studied for using data from multiple sources without sharing them, but they are not usually suitable in applications where each client carries out different tasks. Meanwhile, Transformer has been widely explored in computer vision area due to its capability to learn the common representation through global attention. By leveraging the advantages of Transformer, here we present a new distributed learning framework for multiple image processing tasks, allowing clients to learn distinct tasks with their local data. This arises from a disentangled representation of local and non-local features using a task-specific head/tail and a task-agnostic Vision Transformer. Each client learns a translation from its own task to a common representation using the task-specific networks, while the Transformer body on the server learns global attention between the features embedded in the representation. To enable decomposition between the task-specific and common representations, we propose an alternating training strategy between clients and server. Experimental results on distributed learning for various tasks show that our method synergistically improves the performance of each client with its own data.

DeepPhaseCut: Deep Relaxation in Phase for Unsupervised Fourier Phase Retrieval

Fourier phase retrieval is a classical problem of restoring a signal only from the measured magnitude of its Fourier transform. Although Fienup-type algorithms, which use prior knowledge in both spatial and Fourier domains, have been widely used in practice, they can often stall in local minima. Convex relaxation methods such as PhaseLift and PhaseCut may offer performance guarantees, but these algorithms are usually computationally expensive for practical use. To address this problem, here we propose a novel unsupervised feed-forward neural network for Fourier phase retrieval which generates high quality reconstruction immediately. Unlike the existing deep learning approaches that use a neural network as a regularization term or an end-to-end blackbox model for supervised training, our algorithm is a feed-forward neural network implementation of physics-driven constraints in an unsupervised learning framework. Specifically, our network is composed of two generators: one for the phase estimation using PhaseCut loss, followed by another generator for image reconstruction, all of which are trained simultaneously without matched data. The link to the classical Fienup-type algorithms and the recent symmetry-breaking learning approach is also revealed. Extensive experiments demonstrate that the proposed method outperforms all existing approaches in Fourier phase retrieval problems.

Medical ultrasound image speckle reduction and resolution enhancement using texture compensated multi-resolution convolution neural network

Ultrasound (US) imaging is a mature technology that has widespread applications especially in the healthcare sector. Despite its widespread use and popularity, it has an inherent disadvantage that ultrasound images are prone to speckle and other kinds of noise. The image quality in the low-cost ultrasound imaging systems is degraded due to the presence of such noise and low resolution of such ultrasound systems. Herein, we propose a method for image enhancement where, the overall quality of the US images is improved by simultaneous enhancement of US image resolution and noise suppression. To avoid over-smoothing and preserving structural/texture information, we devise texture compensation in our proposed method to retain the useful anatomical features. Moreover, we also utilize US image formation physics knowledge to generate augmentation datasets which can improve the training of our proposed method. Our experimental results showcase the performance of the proposed network as well as the effectiveness of the utilization of US physics knowledge to generate augmentation datasets.

Wavelet subband discriminator for efficient unsupervised chest X‐ray image restoration

BACKGROUND

Chest X-ray (CXR) images are commonly used to show the internal structure of the human body without invasive intervention. The quality of CXR is an important factor as it affects the accuracy of a clinical diagnosis. Unfortunately, it is difficult to always get good quality CXR scans due to noises and scatters.

PURPOSE

Recently, wavelet directional CycleGAN (WavCycleGAN) has shown promising results in image restoration tasks by removing noise and artifacts without sacrificing high-frequency components of the input image. Unfortunately, WavCycleGAN directly reconstructs wavelet directional images that require a wavelet transform in both the training and test phases, resulting in additional processing steps and unnatural artifacts originating from the wavelet domain image. In addition, WavCycleGAN can only process artifact-related subbands, so it is difficult to apply WavCycleGAN when different levels of artifacts are present in all subbands. To address this, here we present a novel unsupervised CXR image restoration scheme with similar or even better artifact removal performance than WavCycleGAN in spite of wavelet transform being only applied in the training phase.

METHODS

We introduce a novel wavelet subband discriminator which can be combined with CycleGAN or switchable CycleGAN, where wavelet transform is applied only in the training phase for discriminators to match the distribution of wavelet subband components. In our framework, the image restoration network can be still applied in the image domain to prevent unnatural artifacts of the wavelet domain image with the help of the image-domain cycle-consistency loss. In addition, using wavelet subband discriminator makes it possible to remove artifacts in all subbands by utilizing frequency-specific wavelet subband discriminators.

RESULTS

Through extensive experiments for noise and scatter removal in CXRs, we confirm that our method provides competitive performance compared to existing approaches without additional processing steps in the test phase. Furthermore, we show that our wavelet subband discriminator combined with the switchable CycleGAN can provide the flexibility by generating different levels of artifact removal.

CONCLUSIONS

The proposed wavelet subband discriminator can be combined with the existing CycleGAN or switchable CycleGAN structures to construct an efficient unsupervised CXR image reconstruction. The advantage of our wavelet subband discriminator-based CXR image restoration is that, unlike traditional WavCycleGAN, it does not require any additional processing steps in the testing phase and does not generate unnatural artifacts originating from the wavelet domain image. We believe that our wavelet subband discriminator can be applied to various CXR image applications.

Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutation

The widespread application of artificial intelligence in health research is currently hampered by limitations in data availability. Distributed learning methods such as federated learning (FL) and split learning (SL) are introduced to solve this problem as well as data management and ownership issues with their different strengths and weaknesses. The recent proposal of federated split task-agnostic (FESTA) learning tries to reconcile the distinct merits of FL and SL by enabling the multi-task collaboration between participants through Vision Transformer (ViT) architecture, but they suffer from higher communication overhead. To address this, here we present a multi-task distributed learning using ViT with random patch permutation, dubbed p-FESTA. Instead of using a CNN-based head as in FESTA, p-FESTA adopts a simple patch embedder with random permutation, improving the multi-task learning performance without sacrificing privacy. Experimental results confirm that the proposed method significantly enhances the benefit of multi-task collaboration, communication efficiency, and privacy preservation, shedding light on practical multi-task distributed learning in the field of medical imaging.

Diagnosis of coronary layered plaque by deep learning

Background/Introduction

Healed coronary plaques, morphologically characterized by a layered pattern, are signatures of previous plaque disruption and healing. Recent optical coherence tomography (OCT) studies showed that layered plaque is associated with vascular vulnerability and rapid plaque progression. However, the diagnosis of layered plaque requires expertise in OCT image interpretation and is susceptible to interobserver variability.

Purpose

We aimed to develop a deep learning (DL) model for an accurate diagnosis of layered plaque.

Methods

We developed a Visual Transformer (ViT)-based DL model emulating the cardiologists who review consecutive OCT frames to make a diagnosis (Figure 1), and compared it to the standard convolutional neural network (CNN) model. We used 302,415 cross-sectional OCT images from 873 patients collected from 9 sites: 237,021 images from 581 patients for training and internal validation from 8 sites, and 65394 images from 292 patients collected from another site for external validation.

Results

Model performances were evaluated using the area under the receiver operating characteristics (AUC). In the five-fold cross validation, the ViT-based model showed better performance than the standard CNN-based model with AUC of 0.886 (95% confidence interval [CI], 0.882–0.891) compared with 0.797 (95% CI, 0.790–0.804). The ViT-based model also outperformed the standard CNN-based model in the external validation, with an AUC of 0.857 (95% CI, 0.849–0.864) compared to 0.806 (95% CI, 0.797–0.815) (Figure 2).

Conclusion(s)

The ViT-based DL model will help cardiologists to make an accurate diagnosis of layered plaque, which might help to stratify the risk of future adverse cardiac events.

Funding Acknowledgement

Type of funding sources: Private grant(s) and/or Sponsorship. Main funding source(s): Mrs. Gillian Gray through the Allan Gray Fellowship Fund in Cardiology

Optical coherence tomography in coronary atherosclerosis assessment and intervention

Since optical coherence tomography (OCT) was first performed in humans two decades ago, this imaging modality has been widely adopted in research on coronary atherosclerosis and adopted clinically for the optimization of percutaneous coronary intervention. In the past 10 years, substantial advances have been made in the understanding of in vivo vascular biology using OCT. Identification by OCT of culprit plaque pathology could potentially lead to a major shift in the management of patients with acute coronary syndromes. Detection by OCT of healed coronary plaque has been important in our understanding of the mechanisms involved in plaque destabilization and healing with the rapid progression of atherosclerosis. Accurate detection by OCT of sequelae from percutaneous coronary interventions that might be missed by angiography could improve clinical outcomes. In addition, OCT has become an essential diagnostic modality for myocardial infarction with non-obstructive coronary arteries. Insight into neoatherosclerosis from OCT could improve our understanding of the mechanisms of very late stent thrombosis. The appropriate use of OCT depends on accurate interpretation and understanding of the clinical significance of OCT findings. In this Review, we summarize the state of the art in cardiac OCT and facilitate the uniform use of this modality in coronary atherosclerosis. Contributions have been made by clinicians and investigators worldwide with extensive experience in OCT, with the aim that this document will serve as a standard reference for future research and clinical application.

Multi-domain CT translation by a routable translation network

Objective. To unify the style of CT images from multiple sources, we propose a novel multi-domain image translation network to convert CT images from different scan parameters and manufacturers by simply changing a routing vector. Approach. Unlike the existing multi-domain translation techniques, our method is based on a shared encoder and a routable decoder architecture to maximize the expressivity and conditioning power of the network. Main results. Experimental results show that the proposed CT image conversion can minimize the variation of image characteristics caused by imaging parameters, reconstruction algorithms, and hardware designs. Quantitative results and clinical evaluation from radiologists also show that our method can provide accurate translation results. Significance. Quantitative evaluation of CT images from multi-site or longitudinal studies has been a difficult problem due to the image variation depending on CT scan parameters and manufacturers. The proposed method can be utilized to address this for the quantitative analysis of multi-domain CT images.

Low-Dose Sparse-View HAADF-STEM-EDX Tomography of Nanocrystals Using Unsupervised Deep Learning

High-angle annular dark-field (HAADF) scanning transmission electron microscopy (STEM) can be acquired together with energy dispersive X-ray (EDX) spectroscopy to give complementary information on the nanoparticles being imaged. Recent deep learning approaches show potential for accurate 3D tomographic reconstruction for these applications, but a large number of high-quality electron micrographs are usually required for supervised training, which may be difficult to collect due to the damage on the particles from the electron beam. To overcome these limitations and enable tomographic reconstruction even in low-dose sparse-view conditions, here we present an unsupervised deep learning method for HAADF-STEM-EDX tomography. Specifically, to improve the EDX image quality from low-dose condition, a HAADF-constrained unsupervised denoising approach is proposed. Additionally, to enable extreme sparse-view tomographic reconstruction, an unsupervised view enrichment scheme is proposed in the projection domain. Extensive experiments with different types of quantum dots show that the proposed method offers a high-quality reconstruction even with only three projection views recorded under low-dose conditions.

Self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation

Although deep learning-based computer-aided diagnosis systems have recently achieved expert-level performance, developing a robust model requires large, high-quality data with annotations that are expensive to obtain. This situation poses a conundrum that annually-collected chest x-rays cannot be utilized due to the absence of labels, especially in deprived areas. In this study, we present a framework named distillation for self-supervision and self-train learning (DISTL) inspired by the learning process of the radiologists, which can improve the performance of vision transformer simultaneously with self-supervision and self-training through knowledge distillation. In external validation from three hospitals for diagnosis of tuberculosis, pneumothorax, and COVID-19, DISTL offers gradually improved performance as the amount of unlabeled data increase, even better than the fully supervised model with the same amount of labeled data. We additionally show that the model obtained with DISTL is robust to various real-world nuisances, offering better applicability in clinical setting.

Although deep learning-based computer-aided diagnosis systems have recently achieved expert level performance, developing a robust model requires large, high-quality data with annotations. Here, the authors present a framework which can improve the performance of vision transformer simultaneously with self-supervision and self-training.

Deep learning enables reference-free isotropic super-resolution for volumetric fluorescence microscopy

Volumetric imaging by fluorescence microscopy is often limited by anisotropic spatial resolution, in which the axial resolution is inferior to the lateral resolution. To address this problem, we present a deep-learning-enabled unsupervised super-resolution technique that enhances anisotropic images in volumetric fluorescence microscopy. In contrast to the existing deep learning approaches that require matched high-resolution target images, our method greatly reduces the effort to be put into practice as the training of a network requires only a single 3D image stack, without a priori knowledge of the image formation process, registration of training data, or separate acquisition of target data. This is achieved based on the optimal transport-driven cycle-consistent generative adversarial network that learns from an unpaired matching between high-resolution 2D images in the lateral image plane and low-resolution 2D images in other planes. Using fluorescence confocal microscopy and light-sheet microscopy, we demonstrate that the trained network not only enhances axial resolution but also restores suppressed visual details between the imaging planes and removes imaging artifacts.

Volumetric fluorescence microscopy is often limited by anisotropic spatial resolution. Here, the authors present an unsupervised deep-learning approach that enhances axial resolution by learning from high-resolution lateral images, and demonstrate isotropic resolution and restoration of suppressed visual details.

Unsupervised Deep Learning Methods for Biological Image Reconstruction and Enhancement: An overview from a signal processing perspective

Recently, deep learning approaches have become the main research frontier for biological image reconstruction and enhancement problems thanks to their high performance, along with their ultra-fast inference times. However, due to the difficulty of obtaining matched reference data for supervised learning, there has been increasing interest in unsupervised learning approaches that do not need paired reference data. In particular, self-supervised learning and generative models have been successfully used for various biological imaging applications. In this paper, we overview these approaches from a coherent perspective in the context of classical inverse problems, and discuss their applications to biological imaging, including electron, fluorescence and deconvolution microscopy, optical diffraction tomography and functional neuroimaging.

Switchable and Tunable Deep Beamformer Using Adaptive Instance Normalization for Medical Ultrasound

Recent proposals of deep learning-based beamformers for ultrasound imaging (US) have attracted significant attention as computational efficient alternatives to adaptive and compressive beamformers. Moreover, deep beamformers are versatile in that image post-processing algorithms can be readily combined. Unfortunately, with the existing technology, a large number of beamformers need to be trained and stored for different probes, organs, depth ranges, operating frequency, and desired target 'styles', demanding significant resources such as training data, etc. To address this problem, here we propose a switchable and tunable deep beamformer that can switch between various types of outputs such as DAS, MVBF, DMAS, GCF, etc., and also adjust noise removal levels at the inference phase, by using a simple switch or tunable nozzle. This novel mechanism is implemented through Adaptive Instance Normalization (AdaIN) layers, so that distinct outputs can be generated using a single generator by merely changing the AdaIN codes. Experimental results using B-mode focused ultrasound confirm the flexibility and efficacy of the proposed method for various applications.

Deep learning model for diagnosing gastric mucosal lesions using endoscopic images: development, validation, and method comparison

BACKGROUND AND AIMS

Endoscopic differential diagnoses of gastric mucosal lesions (benign gastric ulcer, early gastric cancer [EGC], and advanced gastric cancer) remain challenging. We aimed to develop and validate convolutional neural network-based artificial intelligence (AI) models: lesion detection, differential diagnosis (AI-DDx), and invasion depth (AI-ID; pT1a vs pT1b among EGC) models.

METHODS

This study included 1366 consecutive patients with gastric mucosal lesions from 2 referral centers in Korea. One representative endoscopic image from each patient was used. Histologic diagnoses were set as the criterion standard. Performance of the AI-DDx (training/internal/external validation set, 1009/112/245) and AI-ID (training/internal/external validation set, 620/68/155) was compared with visual diagnoses by independent endoscopists (stratified by novice [<1 year of experience], intermediate [2-3 years of experience], and expert [>5 years of experience]) and EUS results, respectively.

RESULTS

The AI-DDx showed good diagnostic performance for both internal (area under the receiver operating characteristic curve [AUROC] = .86) and external validation (AUROC = .86). The performance of the AI-DDx was better than that of novice (AUROC = .82, P = .01) and intermediate endoscopists (AUROC = .84, P = .02) but was comparable with experts (AUROC = .89, P = .12) in the external validation set. The AI-ID showed a fair performance in both internal (AUROC = .78) and external validation sets (AUROC = .73), which were significantly better than EUS results performed by experts (internal validation, AUROC = .62; external validation, AUROC = .56; both P < .001).

CONCLUSIONS

The AI-DDx was comparable with experts and outperformed novice and intermediate endoscopists for the differential diagnosis of gastric mucosal lesions. The AI-ID performed better than EUS for evaluation of invasion depth.

Radiation Dose Reduction in Digital Mammography by Deep-Learning Algorithm Image Reconstruction: A Preliminary Study

Purpose

To develop a denoising convolutional neural network-based image processing technique and investigate its efficacy in diagnosing breast cancer using low-dose mammography imaging.

Materials and Methods

A total of 6 breast radiologists were included in this prospective study. All radiologists independently evaluated low-dose images for lesion detection and rated them for diagnostic quality using a qualitative scale. After application of the denoising network, the same radiologists evaluated lesion detectability and image quality. For clinical application, a consensus on lesion type and localization on preoperative mammographic examinations of breast cancer patients was reached after discussion. Thereafter, coded low-dose, reconstructed full-dose, and full-dose images were presented and assessed in a random order.

Results

Lesions on 40% reconstructed full-dose images were better perceived when compared with low-dose images of mastectomy specimens as a reference. In clinical application, as compared to 40% reconstructed images, higher values were given on full-dose images for resolution (p < 0.001); diagnostic quality for calcifications (p < 0.001); and for masses, asymmetry, or architectural distortion (p = 0.037). The 40% reconstructed images showed comparable values to 100% full-dose images for overall quality (p = 0.547), lesion visibility (p = 0.120), and contrast (p = 0.083), without significant differences.

Conclusion

Effective denoising and image reconstruction processing techniques can enable breast cancer diagnosis with substantial radiation dose reduction.

Multi-task vision transformer using low-level chest X-ray feature corpus for COVID-19 diagnosis and severity quantification

Developing a robust algorithm to diagnose and quantify the severity of the novel coronavirus disease 2019 (COVID-19) using Chest X-ray (CXR) requires a large number of well-curated COVID-19 datasets, which is difficult to collect under the global COVID-19 pandemic. On the other hand, CXR data with other findings are abundant. This situation is ideally suited for the Vision Transformer (ViT) architecture, where a lot of unlabeled data can be used through structural modeling by the self-attention mechanism. However, the use of existing ViT may not be optimal, as the feature embedding by direct patch flattening or ResNet backbone in the standard ViT is not intended for CXR. To address this problem, here we propose a novel Multi-task ViT that leverages low-level CXR feature corpus obtained from a backbone network that extracts common CXR findings. Specifically, the backbone network is first trained with large public datasets to detect common abnormal findings such as consolidation, opacity, edema, etc. Then, the embedded features from the backbone network are used as corpora for a versatile Transformer model for both the diagnosis and the severity quantification of COVID-19. We evaluate our model on various external test datasets from totally different institutions to evaluate the generalization capability. The experimental results confirm that our model can achieve state-of-the-art performance in both diagnosis and severity quantification tasks with outstanding generalization capability, which are sine qua non of widespread deployment.

Unsupervised CT Metal Artifact Learning Using Attention-Guided β-CycleGAN

Metal artifact reduction (MAR) is one of the most important research topics in computed tomography (CT). With the advance of deep learning approaches for image reconstruction, various deep learning methods have been suggested for metal artifact reduction, among which supervised learning methods are most popular. However, matched metal-artifact-free and metal artifact corrupted image pairs are difficult to obtain in real CT acquisition. Recently, a promising unsupervised learning for MAR was proposed using feature disentanglement, but the resulting network architecture is so complicated that it is difficult to handle large size clinical images. To address this, here we propose a simple and effective unsupervised learning method for MAR. The proposed method is based on a novel β -cycleGAN architecture derived from the optimal transport theory for appropriate feature space disentanglement. Moreover, by adding the convolutional block attention module (CBAM) layers in the generator, we show that the metal artifacts can be more focused so that it can be effectively removed. Experimental results confirm that we can achieve improved metal artifact reduction that preserves the detailed texture of the original image.

Unpaired MR Motion Artifact Deep Learning Using Outlier-Rejecting Bootstrap Aggregation

Recently, deep learning approaches for MR motion artifact correction have been extensively studied. Although these approaches have shown high performance and lower computational complexity compared to classical methods, most of them require supervised training using paired artifact-free and artifact-corrupted images, which may prohibit its use in many important clinical applications. For example, transient severe motion (TSM) due to acute transient dyspnea in Gd-EOB-DTPA-enhanced MR is difficult to control and model for paired data generation. To address this issue, here we propose a novel unpaired deep learning scheme that does not require matched motion-free and motion artifact images. Specifically, the first step of our method is k -space random subsampling along the phase encoding direction that can remove some outliers probabilistically. In the second step, the neural network reconstructs fully sampled resolution image from a downsampled k -space data, and motion artifacts can be reduced in this step. Last, the aggregation step through averaging can further improve the results from the reconstruction network. We verify that our method can be applied for artifact correction from simulated motion as well as real motion from TSM successfully from both single and multi-coil data with and without k -space raw data, outperforming existing state-of-the-art deep learning methods.

Continuous Conversion of CT Kernel Using Switchable CycleGAN With AdaIN

X-ray computed tomography (CT) uses different filter kernels to highlight different structures. Since the raw sinogram data is usually removed after the reconstruction, in case there is additional need for other types of kernel images that were not previously generated, the patient may need to be scanned again. Accordingly, there exists increasing demand for post-hoc image domain conversion from one kernel to another without sacrificing the image quality. In this paper, we propose a novel unsupervised continuous kernel conversion method using cycle-consistent generative adversarial network (cycleGAN) with adaptive instance normalization (AdaIN). Even without paired training data, not only can our network translate the images between two different kernels, but it can also convert images along the interpolation path between the two kernel domains. We also show that the quality of generated images can be further improved if intermediate kernel domain images are available. Experimental results confirm that our method not only enables accurate kernel conversion that is comparable to supervised learning methods, but also generates intermediate kernel images in the unseen domain that are useful for hypopharyngeal cancer diagnosis.

CycleGAN denoising of extreme low-dose cardiac CT using wavelet-assisted noise disentanglement

In electrocardiography (ECG) gated cardiac CT angiography (CCTA), multiple images covering the entire cardiac cycle are taken continuously, so reduction of the accumulated radiation dose could be an important issue for patient safety. Although ECG-gated dose modulation (so-called ECG pulsing) is used to acquire many phases of CT images at a low dose, the reduction of the radiation dose introduces noise into the image reconstruction. To address this, we developed a high performance unsupervised deep learning method using noise disentanglement that can effectively learn the noise patterns even from extreme low dose CT images. For noise disentanglement, we use a wavelet transform to extract the high-frequency signals that contain the most noise. Since matched low-dose and high-dose cardiac CT data are impossible to obtain in practice, our neural network was trained in an unsupervised manner using cycleGAN for the extracted high frequency signals from the low-dose and unpaired high-dose CT images. Once the network is trained, denoised images are obtained by subtracting the estimated noise components from the input images. Image quality evaluation of the denoised images from only 4% dose CT images was performed by experienced radiologists for several anatomical structures. Visual grading analysis was conducted according to the sharpness level, noise level, and structural visibility. Also, the signal-to-noise ratio was calculated. The evaluation results showed that the quality of the images produced by the proposed method is much improved compared to low-dose CT images and to the baseline cycleGAN results. The proposed noise-disentangled cycleGAN with wavelet transform effectively removed noise from extreme low-dose CT images compared to the existing baseline algorithms. It can be an important denoising platform for low-dose CT.

Variational Formulation of Unsupervised Deep Learning for Ultrasound Image Artifact Removal

Recently, deep learning approaches have been successfully used for ultrasound (US) image artifact removal. However, paired high-quality images for supervised training are difficult to obtain in many practical situations. Inspired by the recent theory of unsupervised learning using optimal transport driven CycleGAN (OT-CycleGAN), here, we investigate the applicability of unsupervised deep learning for US artifact removal problems without matched reference data. Two types of OT-CycleGAN approaches are employed: one with the partial knowledge of the image degradation physics and the other with the lack of such knowledge. Various US artifact removal problems are then addressed using the two types of OT-CycleGAN. Experimental results for various unsupervised US artifact removal tasks confirmed that our unsupervised learning method delivers results comparable to supervised learning in many practical applications.

DeepRegularizer: Rapid Resolution Enhancement of Tomographic Imaging Using Deep Learning

Optical diffraction tomography measures the three-dimensional refractive index map of a specimen and visualizes biochemical phenomena at the nanoscale in a non-destructive manner. One major drawback of optical diffraction tomography is poor axial resolution due to limited access to the three-dimensional optical transfer function. This missing cone problem has been addressed through regularization algorithms that use a priori information, such as non-negativity and sample smoothness. However, the iterative nature of these algorithms and their parameter dependency make real-time visualization impossible. In this article, we propose and experimentally demonstrate a deep neural network, which we term DeepRegularizer, that rapidly improves the resolution of a three-dimensional refractive index map. Trained with pairs of datasets (a raw refractive index tomogram and a resolution-enhanced refractive index tomogram via the iterative total variation algorithm), the three-dimensional U-net-based convolutional neural network learns a transformation between the two tomogram domains. The feasibility and generalizability of our network are demonstrated using bacterial cells and a human leukaemic cell line, and by validating the model across different samples. DeepRegularizer offers more than an order of magnitude faster regularization performance compared to the conventional iterative method. We envision that the proposed data-driven approach can bypass the high time complexity of various image reconstructions in other imaging modalities.

Two-stage deep learning for accelerated 3D time-of-flight MRA without matched training data

Time-of-flight magnetic resonance angiography (TOF-MRA) is one of the most widely used non-contrast MR imaging methods to visualize blood vessels, but due to the 3-D volume acquisition highly accelerated acquisition is necessary. Accordingly, high quality reconstruction from undersampled TOF-MRA is an important research topic for deep learning. However, most existing deep learning works require matched reference data for supervised training, which are often difficult to obtain. By extending the recent theoretical understanding of cycleGAN from the optimal transport theory, here we propose a novel two-stage unsupervised deep learning approach, which is composed of the multi-coil reconstruction network along the coronal plane followed by a multi-planar refinement network along the axial plane. Specifically, the first network is trained in the square-root of sum of squares (SSoS) domain to achieve high quality parallel image reconstruction, whereas the second refinement network is designed to efficiently learn the characteristics of highly-activated blood flow using double-headed projection discriminator. Extensive experiments demonstrate that the proposed learning process without matched reference exceeds performance of state-of-the-art compressed sensing (CS)-based method and provides comparable or even better results than supervised learning approaches.

CycleMorph: Cycle consistent unsupervised deformable image registration

Image registration is a fundamental task in medical image analysis. Recently, many deep learning based image registration methods have been extensively investigated due to their comparable performance with the state-of-the-art classical approaches despite the ultra-fast computational time. However, the existing deep learning methods still have limitations in the preservation of original topology during the deformation with registration vector fields. To address this issues, here we present a cycle-consistent deformable image registration, dubbed CycleMorph. The cycle consistency enhances image registration performance by providing an implicit regularization to preserve topology during the deformation. The proposed method is so flexible that it can be applied for both 2D and 3D registration problems for various applications, and can be easily extended to multi-scale implementation to deal with the memory issues in large volume registration. Experimental results on various datasets from medical and non-medical applications demonstrate that the proposed method provides effective and accurate registration on diverse image pairs within a few seconds. Qualitative and quantitative evaluations on deformation fields also verify the effectiveness of the cycle consistency of the proposed method.

Unpaired Training of Deep Learning tMRA for Flexible Spatio-Temporal Resolution

Time-resolved MR angiography (tMRA) has been widely used for dynamic contrast enhanced MRI (DCE-MRI) due to its highly accelerated acquisition. In tMRA, the periphery of the k -space data are sparsely sampled so that neighbouring frames can be merged to construct one temporal frame. However, this view-sharing scheme fundamentally limits the temporal resolution, and it is not possible to change the view-sharing number to achieve different spatio-temporal resolution trade-offs. Although many deep learning approaches have been recently proposed for MR reconstruction from sparse samples, the existing approaches usually require matched fully sampled k -space reference data for supervised training, which is not suitable for tMRA due to the lack of high spatio-temporal resolution ground-truth images. To address this problem, here we propose a novel unpaired training scheme for deep learning using optimal transport driven cycle-consistent generative adversarial network (cycleGAN). In contrast to the conventional cycleGAN with two pairs of generator and discriminator, the new architecture requires just a single pair of generator and discriminator, which makes the training much simpler but still improves the performance. Reconstruction results using in vivo tMRA and simulation data set confirm that the proposed method can immediately generate high quality reconstruction results at various choices of view-sharing numbers, allowing us to exploit better trade-off between spatial and temporal resolution in time-resolved MR angiography.