Noisy-As-Clean: Learning Self-supervised Denoising from Corrupted Image

Object Adaptive Self-Supervised Dense Visual Pre-Training

Self-supervised visual pre-training models have achieved significant success without employing expensive annotations. Nevertheless, most of these models focus on iconic single-instance datasets (e.g. ImageNet), ignoring the insufficient discriminative representation for non-iconic multi-instance datasets (e.g. COCO). In this paper, we propose a novel Object Adaptive Dense Pre-training (OADP) method to learn the visual representation directly on the multi-instance datasets (e.g., PASCAL VOC and COCO) for dense prediction tasks (e.g., object detection and instance segmentation). We present a novel object-aware and learning-adaptive random view augmentation to focus the contrastive learning to enhance the discrimination of object presentations from large to small scale during different learning stages. Furthermore, the representations across different scale and resolutions are integrated so that the method can learn diverse representations. In the experiment, we evaluated OADP pre-trained on PASCAL VOC and COCO. Results show that our method has better performances than most existing state-of-the-art methods when transferring to various downstream tasks, including image classification, object detection, instance segmentation and semantic segmentation.

Investigating self-supervised image denoising with denaturation

Self-supervised learning for image denoising problems in the presence of denaturation for noisy data is a crucial approach in machine learning. However, theoretical understanding of the performance of the approach that uses denatured data is lacking. To provide better understanding of the approach, in this paper, we analyze a self-supervised denoising algorithm that uses denatured data in depth through theoretical analysis and numerical experiments. Through the theoretical analysis, we discuss that the algorithm finds desired solutions to the optimization problem with the population risk, while the guarantee for the empirical risk depends on the hardness of the denoising task in terms of denaturation levels. We also conduct several experiments to investigate the performance of an extended algorithm in practice. The results indicate that the algorithm training with denatured images works, and the empirical performance aligns with the theoretical results. These results suggest several insights for further improvement of self-supervised image denoising that uses denatured data in future directions.

A review of denoising methods in single-particle cryo-EM

Cryo-EM has become a vital technique for resolving macromolecular structures at near-atomic resolution, enabling the visualization of proteins and large molecular complexes. However, the images are often accompanied by extremely low SNR, which poses significant challenges for subsequent processes such as particle picking and 3D reconstruction. Effective denoising methods can substantially improve SNR, making downstream analyzes more accurate and reliable. Thus, image denoising is an essential step in cryo-EM data processing. This paper comprehensively reviews recent advances in image denoising methods for single-particle analysis, covering approaches from traditional filtering methods to the latest deep learning-based strategies. By analyzing and comparing mainstream denoising methods, this review aims to advance the field of single-particle cryo-EM denoising, facilitate the acquisition of higher-quality images, and offer valuable insights for researchers new to the field.

MRI recovery with self-calibrated denoisers without fully-sampled data

OBJECTIVE

Acquiring fully sampled training data is challenging for many MRI applications. We present a self-supervised image reconstruction method, termed ReSiDe, capable of recovering images solely from undersampled data.

MATERIALS AND METHODS

ReSiDe is inspired by plug-and-play (PnP) methods, but unlike traditional PnP approaches that utilize pre-trained denoisers, ReSiDe iteratively trains the denoiser on the image or images that are being reconstructed. We introduce two variations of our method: ReSiDe-S and ReSiDe-M. ReSiDe-S is scan-specific and works with a single set of undersampled measurements, while ReSiDe-M operates on multiple sets of undersampled measurements and provides faster inference. Studies I, II, and III compare ReSiDe-S and ReSiDe-M against other self-supervised or unsupervised methods using data from T1- and T2-weighted brain MRI, MRXCAT digital perfusion phantom, and first-pass cardiac perfusion, respectively.

RESULTS

ReSiDe-S and ReSiDe-M outperform other methods in terms of peak signal-to-noise ratio and structural similarity index measure for Studies I and II, and in terms of expert scoring for Study III.

DISCUSSION

We present a self-supervised image reconstruction method and validate it in both static and dynamic MRI applications. These developments can benefit MRI applications where the availability of fully sampled training data is limited.

Label-free nanoscopy of cell metabolism by ultrasensitive reweighted visible stimulated Raman scattering

Super-resolution imaging of cell metabolism is hindered by the incompatibility of small metabolites with fluorescent dyes and the limited resolution of imaging mass spectrometry. We present ultrasensitive reweighted visible stimulated Raman scattering (URV-SRS), a label-free vibrational imaging technique for multiplexed nanoscopy of intracellular metabolites. We developed a visible SRS microscope with extensive pulse chirping to improve the detection limit to ~4,000 molecules and introduced a self-supervised multi-agent denoiser to suppress non-independent noise in SRS by over 7.2 dB, resulting in a 50-fold sensitivity enhancement over near-infrared SRS. Leveraging the enhanced sensitivity, we employed Fourier reweighting to amplify sub-100-nm spatial frequencies that were previously overwhelmed by noise. Validated by Fourier ring correlation, we achieved a lateral resolution of 86 nm in cell imaging. We visualized the reprogramming of metabolic nanostructures associated with virus replication in host cells and subcellular fatty acid synthesis in engineered bacteria, demonstrating its capability towards nanoscopic spatial metabolomics.

Real-World Low-Dose CT Image Denoising by Patch Similarity Purification

Reducing the radiation dose in CT scanning is important to alleviate the damage to the human health in clinical scenes. A promising way is to replace the normal-dose CT (NDCT) imaging by low-dose CT (LDCT) imaging with lower tube voltage and tube current. This often brings severe noise to the LDCT images, which adversely affects the diagnosis accuracy. Most of existing LDCT image denoising networks are trained either with synthetic LDCT images or real-world LDCT and NDCT image pairs with huge spatial misalignment. However, the synthetic noise is very different from the complex noise in real-world LDCT images, while the huge spatial misalignment brings inaccurate predictions of tissue structures in the denoised LDCT images. To well utilize real-world LDCT and NDCT image pairs for LDCT image denoising, in this paper, we introduce a new Patch Similarity Purification (PSP) strategy to construct high-quality training dataset for network training. Specifically, our PSP strategy first perform binarization for each pair of image patches cropped from the corresponding LDCT and NDCT image pairs. For each pair of binary masks, it then computes their similarity ratio by common mask calculation, and the patch pair can be selected as a training sample if their mask similarity ratio is higher than a threshold. By using our PSP strategy, each training set of our Rabbit and Patient datasets contain hundreds of thousands of real-world LDCT and NDCT image patch pairs with negligible misalignment. Extensive experiments demonstrate the usefulness of our PSP strategy on purifying the training data and the effectiveness of training LDCT image denoising networks on our datasets. The code and dataset are provided at https://github.com/TuTusong/PSP.

Contrastive Learning vs. Self-Learning vs. Deformable Data Augmentation in Semantic Segmentation of Medical Images

To develop a robust segmentation model, encoding the underlying features/structures of the input data is essential to discriminate the target structure from the background. To enrich the extracted feature maps, contrastive learning and self-learning techniques are employed, particularly when the size of the training dataset is limited. In this work, we set out to investigate the impact of contrastive learning and self-learning on the performance of the deep learning-based semantic segmentation. To this end, three different datasets were employed used for brain tumor and hippocampus delineation from MR images (BraTS and Decathlon datasets, respectively) and kidney segmentation from CT images (Decathlon dataset). Since data augmentation techniques are also aimed at enhancing the performance of deep learning methods, a deformable data augmentation technique was proposed and compared with contrastive learning and self-learning frameworks. The segmentation accuracy for the three datasets was assessed with and without applying data augmentation, contrastive learning, and self-learning to individually investigate the impact of these techniques. The self-learning and deformable data augmentation techniques exhibited comparable performance with Dice indices of 0.913 ± 0.030 and 0.920 ± 0.022 for kidney segmentation, 0.890 ± 0.035 and 0.898 ± 0.027 for hippocampus segmentation, and 0.891 ± 0.045 and 0.897 ± 0.040 for lesion segmentation, respectively. These two approaches significantly outperformed the contrastive learning and the original model with Dice indices of 0.871 ± 0.039 and 0.868 ± 0.042 for kidney segmentation, 0.872 ± 0.045 and 0.865 ± 0.048 for hippocampus segmentation, and 0.870 ± 0.049 and 0.860 ± 0.058 for lesion segmentation, respectively. The combination of self-learning with deformable data augmentation led to a robust segmentation model with no outliers in the outcomes. This work demonstrated the beneficial impact of self-learning and deformable data augmentation on organ and lesion segmentation, where no additional training datasets are needed.

Computational ghost imaging enhanced by degradation models for under-sampling

Computational ghost imaging (CGI) allows two-dimensional (2D) imaging by using spatial light modulators and bucket detectors. However, most CGI methods attempt to obtain 2D images through measurements with a single sampling ratio. Here, we propose a CGI method enhanced by degradation models for under-sampling, which can be reflected by results from measurements with different sampling ratios. We utilize results from low-sampling-ratio measurements and normal-sampling-ratio measurements to train the neural network for the degradation model, which is fitted through self-supervised learning. We obtain final results by importing normal-sampling-ratio results into the neural network with optimal parameters. We experimentally demonstrate improved results from the CGI method using degradation models for under-sampling. Our proposed method would promote the development of CGI in many applications.

Self-supervised tomographic image noise suppression via residual image prior network

Computed tomography (CT) denoising is a challenging task in medical imaging that has garnered considerable attention. Supervised networks require a lot of noisy-clean image pairs, which are always unavailable in clinical settings. Existing self-supervised algorithms for suppressing noise with paired noisy images have limitations, such as ignoring the residual between similar image pairs during training and insufficiently learning the spectrum information of images. In this study, we propose a Residual Image Prior Network (RIP-Net) to sufficiently model the residual between the paired similar noisy images. Our approach offers new insights into the field by addressing the limitations of existing methods. We first establish a mathematical theorem clarifying the non-equivalence between similar-image-based self-supervised learning and supervised learning. It helps us better understand the strengths and limitations of self-supervised learning. Secondly, we introduce a novel regularization term to model a low-frequency residual image prior. This can improve the accuracy and robustness of our model. Finally, we design a well-structured denoising network capable of exploring spectrum information while simultaneously sensing context messages. The network has dual paths for modeling high and low-frequency compositions in the raw noisy image. Additionally, context perception modules capture local and global interactions to produce high-quality images. The comprehensive experiments on preclinical photon-counting CT, clinical brain CT, and low-dose CT datasets, demonstrate that our RIP-Net is superior to other unsupervised denoising methods.

Weak signal extraction enabled by deep neural network denoising of diffraction data

The removal or cancellation of noise has wide-spread applications in imaging and acoustics. In applications in everyday life, such as image restoration, denoising may even include generative aspects, which are unfaithful to the ground truth. For scientific use, however, denoising must reproduce the ground truth accurately. Denoising scientific data is further challenged by unknown noise profiles. In fact, such data will often include noise from multiple distinct sources, which substantially reduces the applicability of simulation-based approaches. Here we show how scientific data can be denoised by using a deep convolutional neural network such that weak signals appear with quantitative accuracy. In particular, we study X-ray diffraction and resonant X-ray scattering data recorded on crystalline materials. We demonstrate that weak signals stemming from charge ordering, insignificant in the noisy data, become visible and accurate in the denoised data. This success is enabled by supervised training of a deep neural network with pairs of measured low- and high-noise data. We additionally show that using artificial noise does not yield such quantitatively accurate results. Our approach thus illustrates a practical strategy for noise filtering that can be applied to challenging acquisition problems.

An unsupervised two-step training framework for low-dose computed tomography denoising

BACKGROUND

Although low-dose computed tomography (CT) imaging has been more widely adopted in clinical practice to reduce radiation exposure to patients, the reconstructed CT images tend to have more noise, which impedes accurate diagnosis. Recently, deep neural networks using convolutional neural networks to reduce noise in the reconstructed low-dose CT images have shown considerable improvement. However, they need a large number of paired normal- and low-dose CT images to fully train the network via supervised learning methods.

PURPOSE

To propose an unsupervised two-step training framework for image denoising that uses low-dose CT images of one dataset and unpaired high-dose CT images from another dataset.

METHODS

Our proposed framework trains the denoising network in two steps. In the first training step, we train the network using 3D volumes of CT images and predict the center CT slice from them. This pre-trained network is used in the second training step to train the denoising network and is combined with the memory-efficient denoising generative adversarial network (DenoisingGAN), which further enhances both objective and perceptual quality.

RESULTS

The experimental results on phantom and clinical datasets show superior performance over the existing traditional machine learning and self-supervised deep learning methods, and the results are comparable to the fully supervised learning methods.

CONCLUSIONS

We proposed a new unsupervised learning framework for low-dose CT denoising, convincingly improving noisy CT images from both objective and perceptual quality perspectives. Because our denoising framework does not require physics-based noise models or system-dependent assumptions, our proposed method can be easily reproduced; consequently, it can also be generally applicable to various CT scanners or dose levels.

Self-Supervised Learning for Electroencephalography

Decades of research have shown machine learning superiority in discovering highly nonlinear patterns embedded in electroencephalography (EEG) records compared with conventional statistical techniques. However, even the most advanced machine learning techniques require relatively large, labeled EEG repositories. EEG data collection and labeling are costly. Moreover, combining available datasets to achieve a large data volume is usually infeasible due to inconsistent experimental paradigms across trials. Self-supervised learning (SSL) solves these challenges because it enables learning from EEG records across trials with variable experimental paradigms, even when the trials explore different phenomena. It aggregates multiple EEG repositories to increase accuracy, reduce bias, and mitigate overfitting in machine learning training. In addition, SSL could be employed in situations where there is limited labeled training data, and manual labeling is costly. This article: 1) provides a brief introduction to SSL; 2) describes some SSL techniques employed in recent studies, including EEG; 3) proposes current and potential SSL techniques for future investigations in EEG studies; 4) discusses the cons and pros of different SSL techniques; and 5) proposes holistic implementation tips and potential future directions for EEG SSL practices.

Self-supervised noise modeling and sparsity guided electron tomography volumetric image denoising

Cryo-Electron Tomography (cryo-ET) is a revolutionary technique for visualizing macromolecular structures in near-native states. However, the physical limitations of imaging instruments lead to cryo-ET volumetric images with very low Signal-to-Noise Ratio (SNR) with complex noise, which has a side effect on the downstream analysis of the characteristics of observed macromolecules. Additionally, existing methods for image denoising are difficult to be well generalized to the complex noise in cryo-ET volumes. In this work, we propose a self-supervised deep learning model for cryo-ET volumetric image denoising based on noise modeling and sparsity guidance (NMSG), achieved by learning the noise distribution in noisy cryo-ET volumes and introducing sparsity guidance to ensure smoothness. Firstly, a Generative Adversarial Network (GAN) is utilized to learn noise distribution in cryo-ET volumes and generate noisy volumes pair from single volume. Then, a new loss function is devised to both ensure the recovery of ultrastructure and local smoothness. Experiments are done on five real cryo-ET datasets and three simulated cryo-ET datasets. The comprehensive experimental results demonstrate that our method can perform reliable denoising by training on single noisy volume, achieving better results than state-of-the-art single volume-based methods and competitive with methods trained on large-scale datasets.

Context-aware dynamic filtering network for confocal laser endomicroscopy image denoising

Objective.As an emerging diagnosis technology for gastrointestinal diseases, confocal laser endomicroscopy (CLE) is limited by the physical structure of the fiber bundle, leading to the inevitable production of various forms of noise during the imaging process. However, existing denoising methods based on hand-crafted features inefficiently deal with realistic noise in CLE images. To alleviate this challenge, we proposed context-aware kernel estimation and multi-scale dynamic fusion modules to remove realistic noise in CLE images, including multiplicative and additive white noise.Approach.Specifically, a realistic noise statistics model with random noise specific to CLE data is constructed and further used to develop a self-supervised denoised model without the participation of clean images. Secondly, context-aware kernel estimation, which improves the representation of features by similar learnable region weights, addresses the problem of the non-uniform distribution of noises in CLE images and proposes a lightweight denoised model (CLENet). Thirdly, we have developed a multi-scale dynamic fusion module that decouples and recalibrates features, providing a precise and contextually enriched representation of features. Finally, we integrated two developed modules into a U-shaped backbone to build an efficient denoising network named U-CLENet.Main Results.Both proposed methods achieve comparable or better performance with low computational complexity on two gastrointestinal disease CLE image datasets using the same training benchmark.Significance.The proposed approaches improve the visual quality of unclear CLE images for various stages of tumor development, helping to reduce the rate of misdiagnosis in clinical decision-making and achieve computer graphics-assisted diagnosis.

M-Denoiser: Unsupervised image denoising for real-world optical and electron microscopy data

Real-world microscopy data have a large amount of noise due to the limited light/electron that can be used to capture images. The noise of microscopy data is composed of signal-dependent shot noise and signal-independent read noise, and the Poisson-Gaussian noise model is usually used to describe the noise distribution. Meanwhile, the noise is spatially correlated because of the data acquisition process. Due to the lack of clean ground truth, unsupervised and self-supervised denoising algorithms in computer vision shed new light on tackling such tasks by utilizing paired noisy images or one single noisy image. However, they usually make the assumption that the noise is signal-independent or pixel-wise independent, which contradicts with the actual case. Hence, we propose M-Denoiser for denoising real-world microscopy data in an unsupervised manner. Firstly, the shatter module is used to break the dependency and correlation before denoising. Secondly, a novelly designed unsupervised training loss based on a pair of noisy images is proposed for real-world microscopy data. For evaluation, we train our model on optical and electron microscopy datasets. The experimental results show that M-Denoiser achieves the best performance both quantitatively and qualitatively compared with all the baselines.

Self-trained deep convolutional neural network for noise reduction in CT

Purpose

Supervised deep convolutional neural network (CNN)-based methods have been actively used in clinical CT to reduce image noise. The networks of these methods are typically trained using paired high- and low-quality data from a massive number of patients and/or phantom images. This training process is tedious, and the network trained under a given condition may not be generalizable to patient images acquired and reconstructed under different conditions. We propose a self-trained deep CNN (ST_CNN) method for noise reduction in CT that does not rely on pre-existing training datasets.

Approach

The ST_CNN training was accomplished using extensive data augmentation in the projection domain, and the inference was applied to the data itself. Specifically, multiple independent noise insertions were applied to the original patient projection data to generate multiple realizations of low-quality projection data. Then, rotation augmentation was adopted for both the original and low-quality projection data by applying the rotation angle directly on the projection data so that images were rotated at arbitrary angles without introducing additional bias. A large number of paired low- and high-quality images from the same patient were reconstructed and paired for training the ST_CNN model.

Results

No significant difference was found between the ST_CNN and conventional CNN models in terms of the peak signal-to-noise ratio and structural similarity index measure. The ST_CNN model outperformed the conventional CNN model in terms of noise texture and homogeneity in liver parenchyma as well as better subjective visualization of liver lesions. The ST_CNN may sacrifice the sharpness of vessels slightly compared to the conventional CNN model but without affecting the visibility of peripheral vessels or diagnosis of vascular pathology.

Conclusions

The proposed ST_CNN method trained from the data itself may achieve similar image quality in comparison with conventional deep CNN denoising methods pre-trained on external datasets.

Evaluation of a deep learning-based reconstruction method for denoising and image enhancement of shoulder MRI in patients with shoulder pain

OBJECTIVES

To evaluate the diagnostic performance of an automated reconstruction algorithm combining MR imaging acquired using compressed SENSE (CS) with deep learning (DL) in order to reconstruct denoised high-quality images from undersampled MR images in patients with shoulder pain.

METHODS

Prospectively, thirty-eight patients (14 women, mean age 40.0 ± 15.2 years) with shoulder pain underwent morphological MRI using a pseudo-random, density-weighted k-space scheme with an acceleration factor of 2.5 using CS only. An automated DL-based algorithm (CS DL) was used to create reconstructions of the same k-space data as used for CS reconstructions. Images were analyzed by two radiologists and assessed for pathologies, image quality, and visibility of anatomical landmarks using a 4-point Likert scale.

RESULTS

Overall agreement for the detection of pathologies between the CS DL reconstructions and CS images was substantial to almost perfect (κ 0.95 (95% confidence interval 0.82-1.00)). Image quality and the visibility of the rotator cuff, articular cartilage, and axillary recess were overall rated significantly higher for CS DL images compared to CS (p < 0.03). Contrast-to-noise ratios were significantly higher for cartilage/fluid (CS DL 198 ± 24.3, CS 130 ± 32.2, p = 0.02) and ligament/fluid (CS DL 184 ± 17.3, CS 141 ± 23.5, p = 0.03) and SNR values were significantly higher for ligaments and muscle of the CS DL reconstructions (p < 0.04).

CONCLUSION

Evaluation of shoulder pathologies was feasible using a DL-based algorithm for MRI reconstruction and denoising. In clinical routine, CS DL may be beneficial in particular for reducing image noise and may be useful for the detection and better discrimination of discrete pathologies. Assessment of shoulder pathologies was feasible with improved image quality as well as higher SNR using a compressed sensing deep learning-based framework for image reconstructions and denoising.

KEY POINTS

• Automated deep learning-based reconstructions showed a significant increase in signal-to-noise ratio and contrast-to-noise ratio (p < 0.04) with only a slight increase of reconstruction time of 40 s compared to CS. • All pathologies were accurately detected with no loss of diagnostic information or prolongation of the scan time. • Significant improvements of the image quality as well as the visibility of the rotator cuff, articular cartilage, and axillary recess were detected.

Comparison of Training Strategies for Autoencoder-Based Monochromatic Image Denoising

Monochromatic images are used mainly in cases where the intensity of the received signal is examined. The identification of the observed objects as well as the estimation of intensity emitted by them depends largely on the precision of light measurement in image pixels. Unfortunately, this type of imaging is often affected by noise, which significantly degrades the quality of the results. In order to reduce it, numerous deterministic algorithms are used, with Non-Local-Means and Block-Matching-3D being the most widespread and treated as the reference point of the current state-of-the-art. Our article focuses on the utilization of machine learning (ML) for the denoising of monochromatic images in multiple data availability scenarios, including those with no access to noise-free data. For this purpose, a simple autoencoder architecture was chosen and checked for various training approaches on two large and widely used image datasets: MNIST and CIFAR-10. The results show that the method of training as well as architecture and the similarity of images within the image dataset significantly affect the ML-based denoising. However, even without access to any clear data, the performance of such algorithms is frequently well above the current state-of-the-art; therefore, they should be considered for monochromatic image denoising.

Accelerated MRI using intelligent protocolling and subject-specific denoising applied to Alzheimer's disease imaging

Magnetic Resonance Imaging (MR Imaging) is routinely employed in diagnosing Alzheimer's Disease (AD), which accounts for up to 60-80% of dementia cases. However, it is time-consuming, and protocol optimization to accelerate MR Imaging requires local expertise since each pulse sequence involves multiple configurable parameters that need optimization for contrast, acquisition time, and signal-to-noise ratio (SNR). The lack of this expertise contributes to the highly inefficient utilization of MRI services diminishing their clinical value. In this work, we extend our previous effort and demonstrate accelerated MRI via intelligent protocolling of the modified brain screen protocol, referred to as the Gold Standard (GS) protocol. We leverage deep learning-based contrast-specific image-denoising to improve the image quality of data acquired using the accelerated protocol. Since the SNR of MR acquisitions depends on the volume of the object being imaged, we demonstrate subject-specific (SS) image-denoising. The accelerated protocol resulted in a 1.94 × gain in imaging throughput. This translated to a 72.51% increase in MR Value-defined in this work as the ratio of the sum of median object-masked local SNR values across all contrasts to the protocol's acquisition duration. We also computed PSNR, local SNR, MS-SSIM, and variance of the Laplacian values for image quality evaluation on 25 retrospective datasets. The minimum/maximum PSNR gains (measured in dB) were 1.18/11.68 and 1.04/13.15, from the baseline and SS image-denoising models, respectively. MS-SSIM gains were: 0.003/0.065 and 0.01/0.066; variance of the Laplacian (lower is better): 0.104/-0.135 and 0.13/-0.143. The GS protocol constitutes 44.44% of the comprehensive AD imaging protocol defined by the European Prevention of Alzheimer's Disease project. Therefore, we also demonstrate the potential for AD-imaging via automated volumetry of relevant brain anatomies. We performed statistical analysis on these volumetric measurements of the hippocampus and amygdala from the GS and accelerated protocols, and found that 27 locations were in excellent agreement. In conclusion, accelerated brain imaging with the potential for AD imaging was demonstrated, and image quality was recovered post-acquisition using DL-based image denoising models.

Unsupervised Visual Representation Learning via Multi-Dimensional Relationship Alignment

Recently, contrastive learning based on augmentation invariance and instance discrimination has made great achievements, owing to its excellent ability to learn beneficial representations without any manual annotations. However, the natural similarity among instances conflicts with instance discrimination which treats each instance as a unique individual. In order to explore the natural relationship among instances and integrate it into contrastive learning, we propose a novel approach in this paper, Relationship Alignment (RA for abbreviation), which forces different augmented views of current batch instances to main a consistent relationship with other instances. In order to perform RA effectively in existing contrastive learning framework, we design an alternating optimization algorithm where the relationship exploration step and alignment step are optimized respectively. In addition, we add an equilibrium constraint for RA to avoid the degenerate solution, and introduce the expansion handler to make it approximately satisfied in practice. In order to better capture the complex relationship among instances, we additionally propose Multi-Dimensional Relationship Alignment (MDRA for abbreviation), which aims to explore the relationship from multiple dimensions. In practice, we decompose the final high-dimensional feature space into a cartesian product of several low-dimensional subspaces and perform RA in each subspace respectively. We validate the effectiveness of our approach on multiple self-supervised learning benchmarks and get consistent improvements compared with current popular contrastive learning methods. On the most commonly used ImageNet linear evaluation protocol, our RA obtains significant improvements over other methods, our MDRA gets further improvements based on RA to achieve the best performance. The source code of our approach will be released soon.

Optimal Transport for Unsupervised Denoising Learning

Recently, much progress has been made in unsupervised denoising learning. However, existing methods more or less rely on some assumptions on the signal and/or degradation model, which limits their practical performance. How to construct an optimal criterion for unsupervised denoising learning without any prior knowledge on the degradation model is still an open question. Toward answering this question, this work proposes a criterion for unsupervised denoising learning based on the optimal transport theory. This criterion has favorable properties, e.g., approximately maximal preservation of the information of the signal, whilst achieving perceptual reconstruction. Furthermore, though a relaxed unconstrained formulation is used in practical implementation, we prove that the relaxed formulation in theory has the same solution as the original constrained formulation. Experiments on synthetic and real-world data, including realistic photographic, microscopy, depth, and raw depth images, demonstrate that the proposed method even compares favorably with supervised methods, e.g., approaching the PSNR of supervised methods while having better perceptual quality. Particularly, for spatially correlated noise and realistic microscopy images, the proposed method not only achieves better perceptual quality but also has higher PSNR than supervised methods. Besides, it shows remarkable superiority in harsh practical conditions with complex noise, e.g., raw depth images. Code is available at https://github.com/wangweiSJTU/OTUR.

Acquisition of a single grid-based phase-contrast X-ray image using instantaneous frequency and noise filtering

BACKGROUND

To obtain phase-contrast X-ray images, single-grid imaging systems are effective, but Moire artifacts remain a significant issue. The solution for removing Moire artifacts from an image is grid rotation, which can distinguish between these artifacts and sample information within the Fourier space. However, the mechanical movement of grid rotation is slower than the real-time change in Moire artifacts. Thus, Moire artifacts generated during real-time imaging cannot be removed using grid rotation. To overcome this problem, we propose an effective method to obtain phase-contrast X-ray images using instantaneous frequency and noise filtering.

RESULT

The proposed phase-contrast X-ray image using instantaneous frequency and noise filtering effectively suppressed noise with Moire patterns. The proposed method also preserved the clear edge of the inner and outer boundaries and internal anatomical information from the biological sample, outperforming conventional Fourier analysis-based methods, including absorption, scattering, and phase-contrast X-ray images. In particular, when comparing the phase information for the proposed method with the x-axis gradient image from the absorption image, the proposed method correctly distinguished two different types of soft tissue and the detailed information, while the latter method did not.

CONCLUSION

This study successfully achieved a significant improvement in image quality for phase-contrast X-ray images using instantaneous frequency and noise filtering. This study can provide a foundation for real-time bio-imaging research using three-dimensional computed tomography.

Self-Supervised Learning for RGB-Guided Depth Enhancement by Exploiting the Dependency Between RGB and Depth

Due to the imaging mechanism of time-of-flight (ToF) sensors, the captured depth images usually suffer from severe noise and degradation. Though many RGB-guided methods have been proposed for depth image enhancement in the past few years, yet the enhancement performance on real-world depth images is still largely unsatisfactory. Two main reasons are the complexity of realistic noise and degradation in depth images, and the difficulty in collecting noise-clean pairs for supervised enhancement learning. This work aims to develop a self-supervised learning method for RGB-guided depth image enhancement, which does not require any noisy-clean pairs but can significantly boost the enhancement performance on real-world noisy depth images. To this end, we exploit the dependency between RGB and depth images to self-supervise the learning of the enhancement model. It is achieved by maximizing the cross-modal dependency between RGB and depth to promote the enhanced depth having dependency with the RGB of the same scene as much as possible. Furthermore, we augment the cross-modal dependency maximization formulation based on the optimal transport theory to achieve further performance improvement. Experimental results on both synthetic and real-world data demonstrate that our method can significantly outperform existing state-of-the-art methods on depth denoising, multi-path interference suppression, and hole filling. Particularly, our method shows remarkable superiority over existing ones on real-world data in handling various realistic complex degradation. Code is available at https://github.com/wjcyt/SRDE.

Isotropic reconstruction for electron tomography with deep learning

Cryogenic electron tomography (cryoET) allows visualization of cellular structures in situ. However, anisotropic resolution arising from the intrinsic "missing-wedge" problem has presented major challenges in visualization and interpretation of tomograms. Here, we have developed IsoNet, a deep learning-based software package that iteratively reconstructs the missing-wedge information and increases signal-to-noise ratio, using the knowledge learned from raw tomograms. Without the need for sub-tomogram averaging, IsoNet generates tomograms with significantly reduced resolution anisotropy. Applications of IsoNet to three representative types of cryoET data demonstrate greatly improved structural interpretability: resolving lattice defects in immature HIV particles, establishing architecture of the paraflagellar rod in Eukaryotic flagella, and identifying heptagon-containing clathrin cages inside a neuronal synapse of cultured cells. Therefore, by overcoming two fundamental limitations of cryoET, IsoNet enables functional interpretation of cellular tomograms without sub-tomogram averaging. Its application to high-resolution cellular tomograms should also help identify differently oriented complexes of the same kind for sub-tomogram averaging.

Attention mechanism and mixup data augmentation for classification of COVID-19 Computed Tomography images

The Coronavirus disease is quickly spreading all over the world and the emergency situation is still out of control. Latest achievements of deep learning algorithms suggest the use of deep Convolutional Neural Network to implement a computer-aided diagnostic system for automatic classification of COVID-19 CT images. In this paper, we propose to employ a feature-wise attention layer in order to enhance the discriminative features obtained by convolutional networks. Moreover, the original performance of the network has been improved using the mixup data augmentation technique. This work compares the proposed attention-based model against the stacked attention networks, and traditional versus mixup data augmentation approaches. We deduced that feature-wise attention extension, while outperforming the stacked attention variants, achieves remarkable improvements over the baseline convolutional neural networks. That is, ResNet50 architecture extended with a feature-wise attention layer obtained 95.57% accuracy score, which, to best of our knowledge, fixes the state-of-the-art in the challenging COVID-CT dataset.

CERL: A Unified Optimization Framework for Light Enhancement With Realistic Noise

Low-light images captured in the real world are inevitably corrupted by sensor noise. Such noise is spatially variant and highly dependent on the underlying pixel intensity, deviating from the oversimplified assumptions in conventional denoising. Existing light enhancement methods either overlook the important impact of real-world noise during enhancement, or treat noise removal as a separate pre- or post-processing step. We present Coordinated Enhancement for Real-world Low-light Noisy Images (CERL), that seamlessly integrates light enhancement and noise suppression parts into a unified and physics-grounded optimization framework. For the real low-light noise removal part, we customize a self-supervised denoising model that can easily be adapted without referring to clean ground-truth images. For the light enhancement part, we also improve the design of a state-of-the-art backbone. The two parts are then joint formulated into one principled plug-and-play optimization. Our approach is compared against state-of-the-art low-light enhancement methods both qualitatively and quantitatively. Besides standard benchmarks, we further collect and test on a new realistic low-light mobile photography dataset (RLMP), whose mobile-captured photos display heavier realistic noise than those taken by high-quality cameras. CERL consistently produces the most visually pleasing and artifact-free results across all experiments. Our RLMP dataset and codes are available at: https://github.com/VITA-Group/CERL.

Neighbor2Neighbor: A Self-Supervised Framework for Deep Image Denoising

In recent years, image denoising has benefited a lot from deep neural networks. However, these models need large amounts of noisy-clean image pairs for supervision. Although there have been attempts in training denoising networks with only noisy images, existing self-supervised algorithms suffer from inefficient network training, heavy computational burden, or dependence on noise modeling. In this paper, we proposed a self-supervised framework named Neighbor2Neighbor for deep image denoising. We develop a theoretical motivation and prove that by designing specific samplers for training image pairs generation from only noisy images, we can train a self-supervised denoising network similar to the network trained with clean images supervision. Besides, we propose a regularizer in the perspective of optimization to narrow the optimization gap between the self-supervised denoiser and the supervised denoiser. We present a very simple yet effective self-supervised training scheme based on the theoretical understandings: training image pairs are generated by random neighbor sub-samplers, and denoising networks are trained with a regularized loss. Moreover, we propose a training strategy named BayerEnsemble to adapt the Neighbor2Neighbor framework in raw image denoising. The proposed Neighbor2Neighbor framework can enjoy the progress of state-of-the-art supervised denoising networks in network architecture design. It also avoids heavy dependence on the assumption of the noise distribution. We evaluate the Neighbor2Neighbor framework through extensive experiments, including synthetic experiments with different noise distributions and real-world experiments under various scenarios. The code is available online: https://github.com/TaoHuang2018/Neighbor2Neighbor.

Noise2Kernel: Adaptive Self-Supervised Blind Denoising Using a Dilated Convolutional Kernel Architecture

With the advent of unsupervised learning, efficient training of a deep network for image denoising without pairs of noisy and clean images has become feasible. Most current unsupervised denoising methods are built on self-supervised loss with the assumption of zero-mean noise under the signal-independent condition, which causes brightness-shifting artifacts on unconventional noise statistics (i.e., different from commonly used noise models). Moreover, most blind denoising methods require a random masking scheme for training to ensure the invariance of the denoising process. In this study, we propose a dilated convolutional network that satisfies an invariant property, allowing efficient kernel-based training without random masking. We also propose an adaptive self-supervision loss to increase the tolerance for unconventional noise, which is specifically effective in removing salt-and-pepper or hybrid noise where prior knowledge of noise statistics is not readily available. We demonstrate the efficacy of the proposed method by comparing it with state-of-the-art denoising methods using various examples.

MRI RECOVERY WITH A SELF-CALIBRATED DENOISER

Plug-and-play (PnP) methods that employ application-specific denoisers have been proposed to solve inverse problems, including MRI reconstruction. However, training application-specific denoisers is not feasible for many applications due to the lack of training data. In this work, we propose a PnP-inspired recovery method that does not require data beyond the single, incomplete set of measurements. The proposed self-supervised method, called recovery with a self-calibrated denoiser (ReSiDe), trains the denoiser from the patches of the image being recovered. The denoiser training and a call to the denoising subroutine are performed in each iteration of a PnP algorithm, leading to a progressive refinement of the reconstructed image. For validation, we compare ReSiDe with a compressed sensing-based method and a PnP method with BM3D denoising using single-coil MRI brain data.

Image Semantic Analysis in Visual Media Art Using Machine Learning and Neural Machine Translation

The current archaeological work in China has the problems of high cost, large material consumption, more attention on human detection and long time-consuming. It is urgent to use modern high-precision detection technology for auxiliary work. This exploration will also use the semantic recognition system based on deep learning and neural network for the recognition of oracle bone inscriptions in archaeology. Therefore, combined with the concept of multimedia semantic recognition and analysis, a unified real-time target detection semantic analysis model named You Only Look Once (YOLOv2) is established based on the deep convolutional neural network under deep learning in the field of machine learning, to test the semantic analysis of oracle bone inscriptions. Moreover, Faster Region-Convolutional Neural Network (Faster R-CNN) and traditional YOLO models are selected to conduct the controlled experiments. A YOLOv2 recognition system is established based on Diffusion-Convolutional Neural Networks (DCNN) under deep learning. First, the concept and performance of DCNN are studied. Next, the basic information of oracle bone inscriptions is analyzed. A recognition system based on DCNN is established. On the premise that the three models can identify different directions of the same oracle bone inscriptions sample, the detection accuracy and detection loss value of YOLOv2 are better than those of the other two models, the detection accuracy is as high as 0.90, and the loss value is less than 0.10. Therefore, it is considered that this YOLOv2 semantic analysis model can be applied in oracle bone inscriptions and other archaeological work, which improves the work efficiency and simplifies the human work items for the domestic archaeological work. This semantic analysis model is of great help to the pattern recognition of cultural relics in archaeological work, and can help professionals analyze the meaning of patterns faster when there are massive similar oracle bone inscriptions.

Self-Supervised Denoising Image Filter Based on Recursive Deep Neural Network Structure

The purpose of this paper is to propose a novel noise removal method based on deep neural networks that can remove various types of noise without paired noisy and clean data. Because this type of filter generally has relatively poor performance, the proposed noise-to-blur-estimated clean (N2BeC) model introduces a stage-dependent loss function and a recursive learning stage for improved denoised image quality. The proposed loss function regularizes the existing loss function so that the proposed model can better learn image details. Moreover, the recursive learning stage provides the proposed model with an additional opportunity to learn image details. The overall deep neural network consists of three learning stages and three corresponding loss functions. We determine the essential hyperparameters via several simulations. Consequently, the proposed model showed more than 1 dB superior performance compared with the existing noise-to-blur model.

Evaluation of MRI Denoising Methods Using Unsupervised Learning

In this paper we evaluate two unsupervised approaches to denoise Magnetic Resonance Images (MRI) in the complex image space using the raw information that k-space holds. The first method is based on Stein’s Unbiased Risk Estimator, while the second approach is based on a blindspot network, which limits the network’s receptive field. Both methods are tested on two different datasets, one containing real knee MRI and the other consists of synthetic brain MRI. These datasets contain information about the complex image space which will be used for denoising purposes. Both networks are compared against a state-of-the-art algorithm, Non-Local Means (NLM) using quantitative and qualitative measures. For most given metrics and qualitative measures, both networks outperformed NLM, and they prove to be reliable denoising methods.

Weakly-supervised progressive denoising with unpaired CT images

Although low-dose CT imaging has attracted a great interest due to its reduced radiation risk to the patients, it suffers from severe and complex noise. Recent fully-supervised methods have shown impressive performances on CT denoising task. However, they require a huge amount of paired normal-dose and low-dose CT images, which is generally unavailable in real clinical practice. To address this problem, we propose a weakly-supervised denoising framework that generates paired original and noisier CT images from unpaired CT images using a physics-based noise model. Our denoising framework also includes a progressive denoising module that bypasses the challenges of mapping from low-dose to normal-dose CT images directly via progressively compensating the small noise gap. To quantitatively evaluate diagnostic image quality, we present the noise power spectrum and signal detection accuracy, which are well correlated with the visual inspection. The experimental results demonstrate that our method achieves remarkable performances, even superior to fully-supervised CT denoising with respect to the signal detectability. Moreover, our framework increases the flexibility in data collection, allowing us to utilize any unpaired data at any dose levels.

JCS: An Explainable COVID-19 Diagnosis System by Joint Classification and Segmentation

Recently, the coronavirus disease 2019 (COVID-19) has caused a pandemic disease in over 200 countries, influencing billions of humans. To control the infection, identifying and separating the infected people is the most crucial step. The main diagnostic tool is the Reverse Transcription Polymerase Chain Reaction (RT-PCR) test. Still, the sensitivity of the RT-PCR test is not high enough to effectively prevent the pandemic. The chest CT scan test provides a valuable complementary tool to the RT-PCR test, and it can identify the patients in the early-stage with high sensitivity. However, the chest CT scan test is usually time-consuming, requiring about 21.5 minutes per case. This paper develops a novel Joint Classification and Segmentation (JCS) system to perform real-time and explainable COVID- 19 chest CT diagnosis. To train our JCS system, we construct a large scale COVID- 19 Classification and Segmentation (COVID-CS) dataset, with 144,167 chest CT images of 400 COVID- 19 patients and 350 uninfected cases. 3,855 chest CT images of 200 patients are annotated with fine-grained pixel-level labels of opacifications, which are increased attenuation of the lung parenchyma. We also have annotated lesion counts, opacification areas, and locations and thus benefit various diagnosis aspects. Extensive experiments demonstrate that the proposed JCS diagnosis system is very efficient for COVID-19 classification and segmentation. It obtains an average sensitivity of 95.0% and a specificity of 93.0% on the classification test set, and 78.5% Dice score on the segmentation test set of our COVID-CS dataset. The COVID-CS dataset and code are available at https://github.com/yuhuan-wu/JCS.