Channel attention generative adversarial network for super-resolution of glioma magnetic resonance image

BACKGROUND AND OBJECTIVE

Glioma is the most common primary craniocerebral tumor caused by the cancelation of glial cells in the brain and spinal cord, with a high incidence and cure rate. Magnetic resonance imaging (MRI) is a common technique for detecting and analyzing brain tumors. Due to improper hardware and operation, the obtained brain MRI images are low-resolution, making it difficult to detect and grade gliomas accurately. However, super-resolution reconstruction technology can improve the clarity of MRI images and help experts accurately detect and grade glioma.

METHODS

We propose a glioma magnetic resonance image super-resolution reconstruction method based on channel attention generative adversarial network (CGAN). First, we replace the base block of SRGAN with a residual dense block based on the channel attention mechanism. Second, we adopt a relative average discriminator to replace the discriminator in standard GAN. Finally, we add the mean squared error loss to the training, consisting of the mean squared error loss, the L1 norm loss, and the generator's adversarial loss to form the generator loss function.

RESULTS

On the Set5, Set14, Urban100, and glioma datasets, compared with the state-of-the-art algorithms, our proposed CGAN method has improved peak signal-to-noise ratio and structural similarity, and the reconstructed glioma images are more precise than other algorithms.

CONCLUSION

The experimental results show that our CGAN method has apparent improvements in objective evaluation indicators and subjective visual effects, indicating its effectiveness and superiority.

A super-resolution network using channel attention retention for pathology images

Image super-resolution (SR) significantly improves the quality of low-resolution images, and is widely used for image reconstruction in various fields. Although the existing SR methods have achieved distinguished results in objective metrics, most methods focus on real-world images and employ large and complex network structures, which are inefficient for medical diagnosis scenarios. To address the aforementioned issues, the distinction between pathology images and real-world images was investigated, and an SR Network with a wider and deeper attention module called Channel Attention Retention is proposed to obtain SR images with enhanced high-frequency features. This network captures contextual information within and across blocks via residual skips and balances the performance and efficiency by controlling the number of blocks. Meanwhile, a new linear loss was introduced to optimize the network. To evaluate the work and compare multiple SR works, a benchmark dataset bcSR was created, which forces a model training on wider and more critical regions. The results show that the proposed model outperforms state-of-the-art methods in both performance and efficiency, and the newly created dataset significantly improves the reconstruction quality of all compared models. Moreover, image classification experiments demonstrate that the suggested network improves the performance of downstream tasks in medical diagnosis scenarios. The proposed network and dataset provide effective priors for the SR task of pathology images, which significantly improves the diagnosis of relevant medical staff. The source code and the dataset are available on https://github.com/MoyangSensei/CARN-Pytorch.

Advanced Imaging Approaches to Reveal Molecular Mechanisms Governing Neuroendocrine Secretion

Identification of the molecular mechanisms governing neuroendocrine secretion and resulting intercellular communication is one of the great challenges of cell biology to better understand organism physiology and neurosecretion disruption-related pathologies such as hypertension, neurodegenerative, or metabolic diseases. To visualize molecule distribution and dynamics at the nanoscale, many imaging approaches have been developed and are still emerging. In this review, we provide an overview of the pioneering studies using transmission electron microscopy, atomic force microscopy, total internal reflection microscopy, and super-resolution microscopy in neuroendocrine cells to visualize molecular mechanisms driving neurosecretion processes, including exocytosis and associated fusion pores, endocytosis and associated recycling vesicles, and protein-protein or protein-lipid interactions. Furthermore, the potential and the challenges of these different advanced imaging approaches for application in the study of neuroendocrine cell biology are discussed, aiming to guide researchers to select the best approach for their specific purpose around the crucial but not yet fully understood neurosecretion process.

High- and Super-Resolution Imaging of Cell-Cell Interfaces

Physical interfaces mediate interactions between multiple types of cells. Despite the importance of such interfaces to the cells' function, their high-resolution optical imaging has been typically limited due to poor alignment of the interfaces relative to the optical plane of imaging. Here, we present a simple and robust method to align cell-cell interfaces in parallel to the coverslip by adhering the interacting cells to two opposing coverslips and bringing them into contact in a controlled and stable fashion. We demonstrate aberration-free high-resolution imaging of interfaces between live T cells and antigen-presenting cells, known as immune synapses, as an outstanding example. Imaging methods may include multiple diffraction-limited and super-resolution microscopy techniques (e.g., bright-field, confocal, STED, and dSTORM). Thus, our simple and widely compatible approach allows imaging with high- and super-resolution the intricate structure and molecular organization within a variety of cell-cell interfaces.

STORM Microscopy and Cluster Analysis for PcG Studies

Advanced microscopy techniques (such as STORM, STED, and SIM) have recently allowed the visualization of biological samples beyond the diffraction limit of light. Thanks to this breakthrough, the organization of molecules can be revealed within single cells as never before.Here, we describe the application of STochastic Optical Reconstruction Microscopy (STORM) for the study of polycomb group of proteins (PcG) in the context of chromatin organization. We present a clustering algorithm to quantitatively analyze the spatial distribution of nuclear molecules (e.g., EZH2 or its associated chromatin mark H3K27me3) imaged by 2D STORM. This distance-based analysis uses x-y coordinates of STORM localizations to group them into "clusters." Clusters are classified as singles if isolated or into islands if they form a group of closely associated clusters. For each cluster, the algorithm calculates the number of localizations, the area, and the distance to the closest cluster.This approach can be used for every type of adherent cell line and allows the imaging of every protein for which an antibody is available. It represents a comprehensive strategy to visualize and quantify how PcG proteins and related histone marks organize in the nucleus at nanometric resolution.

Super-resolution microscopy reveals the number and distribution of topoisomerase IIα and CENH3 molecules within barley metaphase chromosomes

Topoisomerase IIα (Topo IIα) and the centromere-specific histone H3 variant CENH3 are key proteins involved in chromatin condensation and centromere determination, respectively. Consequently, they are required for proper chromosome segregation during cell divisions. We combined two super-resolution techniques, structured illumination microscopy (SIM) to co-localize Topo IIα and CENH3, and photoactivated localization microscopy (PALM) to determine their molecule numbers in barley metaphase chromosomes. We detected a dispersed Topo IIα distribution along chromosome arms but an accumulation at centromeres, telomeres, and nucleolus-organizing regions. With a precision of 10-50 nm, we counted ~ 20,000-40,000 Topo IIα molecules per chromosome, 28% of them within the (peri)centromere. With similar precision, we identified ~13,500 CENH3 molecules per centromere where Topo IIα proteins and CENH3-containing chromatin intermingle. In short, we demonstrate PALM as a useful method to count and localize single molecules with high precision within chromosomes. The ultrastructural distribution and the detected amount of Topo IIα and CENH3 are instrumental for a better understanding of their functions during chromatin condensation and centromere determination.

Advanced Topics in Quantitative Acoustic Microscopy

Quantitative acoustic microscopy (QAM) reconstructs two-dimensional (2D) maps of the acoustic properties of thin tissue sections. Using ultrahigh frequency transducers (≥ 100 MHz), unstained, micron-thick tissue sections affixed to glass are raster scanned to collect radiofrequency (RF) echo data and generate parametric maps with resolution approximately equal to the ultrasound wavelength. 2D maps of speed of sound, mass density, acoustic impedance, bulk modulus, and acoustic attenuation provide unique and quantitative information that is complementary to typical optical microscopy modalities. Consequently, many biomedical researchers have great interest in utilizing QAM instruments to investigate the acoustic and biomechanical properties of tissues at the micron scale. Unfortunately, current state-of-the-art QAM technology is costly, requires operation by a trained user, and is accompanied by substantial experimental challenges, many of which become more onerous as the transducer frequency is increased. In this chapter, typical QAM technology and standard image formation methods are reviewed. Then, novel experimental and signal processing approaches are presented with the specific goal of reducing QAM instrument costs and improving ease of use. These methods rely on modern techniques based on compressed sensing and sparsity-based deconvolution methods. Together, these approaches could serve as the basis of the next generation of QAM instruments that are affordable and provide high-resolution QAM images with turnkey solutions requiring nearly no training to operate.

Super-Resolution Spinning-Disk Confocal Microscopy Using Optical Photon Reassignment (SoRa) to Visualize the Actin Cytoskeleton in Macrophages

Macrophages are motile, morphologically plastic cells that undergo substantial cytoskeletal remodeling to facilitate their roles in innate and adaptive immunity. Macrophages are adept at producing a variety of specialized actin-driven structures and processes including the formation of podosomes and the ability to engulf particles through phagocytosis and sample large amounts of extracellular fluid via micropinocytosis. Here, we describe techniques for immunostaining proteins and transfecting macrophages with plasmids for use with either fixed or live cell imaging. Furthermore, we discuss the use of spinning-disk super-resolution using optical reassignment to generate sub-diffraction limited structures using this type of confocal microscope.

Expansion Microscopy of Mouse Photoreceptor Cilia

The small size of ciliary structures that underlies photoreceptor function and inherited ciliopathies requires imaging techniques adapted to visualizing them at the highest possible resolution. In addition to powerful super-resolution imaging modalities, emerging approaches to sample preparation, including expansion microscopy (ExM), can provide a robust route to imaging specific molecules at the nanoscale level in the retina. We describe a protocol for applying ExM to whole retinas in order to achieve nanoscale fluorescence imaging of ciliary markers, including tubulin, CEP290, centrin, and CEP164. The results are consistent with those from other super-resolution fluorescence techniques and reveal new insights into their arrangements with respect to the subcompartments of photoreceptor cilia. This technique is complimentary to other imaging modalities used in retinal imaging, and can be carried out in virtually any laboratory, without the need for expensive specialized equipment.

Slow-Off-Rate-Modified Aptamer Labeling for Fluorescence Microscopy and DNA-PAINT

Super-resolution microscopy methods enable the visualization of biological processes on the level of a few nanometers. However, the application of these techniques in biological systems is limited by the availability of small affinity reagents. Slow off-rate-modified aptamers as nucleic acid analogues to antibodies have been successfully applied to improve the resolution and quantification of DNA-PAINT. In this chapter, we describe a protocol for using SOMAmers as labeling reagents for super-resolution microscopy.

Metallic support films reduce optical heating in cryogenic correlative light and electron tomography

Super-resolved cryogenic correlative light and electron tomography is an emerging method that provides both the single-molecule sensitivity and specificity of fluorescence imaging, and the molecular scale resolution and detailed cellular context of tomography, all in vitrified cells preserved in their native hydrated state. Technical hurdles that limit these correlative experiments need to be overcome for the full potential of this approach to be realized. Chief among these is sample heating due to optical excitation which leads to devitrification, a phase transition from amorphous to crystalline ice. Here we show that much of this heating is due to the material properties of the support film of the electron microscopy grid, specifically the absorptivity and thermal conductivity. We demonstrate through experiment and simulation that the properties of the standard holey carbon electron microscopy grid lead to substantial heating under optical excitation. In order to avoid devitrification, optical excitation intensities must be kept orders of magnitude lower than the intensities commonly employed in room temperature super-resolution experiments. We further show that the use of metallic films, either holey gold grids, or custom made holey silver grids, alleviate much of this heating. For example, the holey silver grids permit 20× the optical intensities used on the standard holey carbon grids. Super-resolution correlative experiments conducted on holey silver grids under these increased optical excitation intensities have a corresponding increase in the rate of single-molecule fluorescence localizations. This results in an increased density of localizations and improved correlative imaging without deleterious effects from sample heating.

CT image super-resolution reconstruction based on global hybrid attention

Computer tomography (CT) has played an essential role in the field of medical diagnosis, but the blurry edges and unclear textures in traditional CT images usually interfere the subsequent judgement from radiologists or clinicians. Deep learning based image super-resolution methods have been applied for CT image restoration recently. However, different levels of information of CT image details are mixed and difficult to be mapped from deep features by traditional convolution operations. Moreover, features representing regions of interest (ROIs) in CT images are treated equally as those for background, resulting in low concentration of meaningful features and high redundancy of computation. To tackle these issues, a CT image super-resolution network is proposed based on hybrid attention mechanism and global feature fusion, which consists of the following three parts: 1) stacked Swin Transformer blocks are used as the backbone to extract initial features from the degraded CT image; 2) a multi-branch hierarchical self-attention module (MHSM) is proposed to adaptively map multi-level features representing different levels of image information from the initial features and establish the relationship between these features through a self-attention mechanism, where three branches apply different strategies of integrating convolution, down-sampling and up-sampling operations according to three different scale factors; 3) a multidimensional local topological feature enhancement module (MLTEM) is proposed and plugged into the end of the backbone to refine features in the channel and spatial dimension simultaneously, so that the features representing ROIs could be enhanced while meaningless ones eliminated. Experimental results demonstrate that our method outperform the state-of-the-art super-resolution methods on restoring CT images with respect to peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indices.

Dual attention mechanism network for lung cancer images super-resolution

BACKGROUND AND OBJECTIVE

Currently, the morbidity and mortality of lung cancer rank first among malignant tumors worldwide. Improving the resolution of thin-slice CT of the lung is particularly important for the early diagnosis of lung cancer screening.

METHODS

Aiming at the problems of network training difficulty and low utilization of feature information caused by the deepening of network layers in super-resolution (SR) reconstruction technology, we propose the dual attention mechanism network for single image super-resolution (SISR). Firstly, the feature of a low-resolution image is extracted directly to retain the feature information. Secondly, several independent dual attention mechanism modules are constructed to extract high-frequency details. The introduction of residual connections can effectively solve the gradient disappearance caused by network deepening, and long and short skip connections can effectively enhance the data features. Furthermore, a hybrid loss function speeds up the network's convergence and improves image SR restoration ability. Finally, through the upsampling operation, the reconstructed high-resolution image is obtained.

RESULTS

The results on the Set5 dataset for 4 × enlargement show that compared with traditional SR methods such as Bicubic, VDSR, and DRRN, the average PSNR/SSIM is increased by 3.33 dB / 0.079, 0.41 dB / 0.007 and 0.22 dB / 0.006 respectively. The experimental data fully show that DAMN can better restore the image contour features, obtain higher PSNR, SSIM, and better visual effect.

CONCLUSION

Through the DAMN reconstruction method, the image quality can be improved without increasing radiation exposure and scanning time. Radiologists can enhance their confidence in diagnosing early lung cancer, provide a basis for clinical experts to choose treatment plans, formulate follow-up strategies, and benefit patients in the early stage.

Improved generative adversarial network for retinal image super-resolution

BACKGROUND AND OBJECTIVE

The retina is the only organ in the body that can use visible light for non-invasive observation. By analyzing retinal images, we can achieve early screening, diagnosis and prevention of many ophthalmological and systemic diseases, helping patients avoid the risk of blindness. Due to the powerful feature extraction capabilities, many deep learning super-resolution reconstruction networks have been applied to retinal image analysis and achieved excellent results.

METHODS

Given the lack of high-frequency information and poor visual perception in the current reconstruction results of super-resolution reconstruction networks under large-scale factors, we present an improved generative adversarial network (IGAN) algorithm for retinal image super-resolution reconstruction. Firstly, we construct a novel residual attention block, improving the reconstruction results lacking high-frequency information and texture details under large-scale factors. Secondly, we remove the Batch Normalization layer that affects the quality of image generation in the residual network. Finally, we use the more robust Charbonnier loss function instead of the mean square error loss function and the TV regular term to smooth the training results.

RESULTS

Experimental results show that our proposed method significantly improves objective evaluation indicators such as peak signal-to-noise ratio and structural similarity. The obtained image has rich texture details and a better visual experience than the state-of-the-art image super-resolution methods.

CONCLUSION

Our proposed method can better learn the mapping relationship between low-resolution and high-resolution retinal images. This method can be effectively and stably applied to the analysis of retinal images, providing an effective basis for early clinical treatment.

Structural connectivity mapping in human hippocampal-subfields using super-resolution hybrid diffusion imaging: a feasibility study

PURPOSE

The goal of the current study was to introduce a new methodology that holds a promise to be used in hippocampus-aging studies using sub-millimeter super-resolution hybrid diffusion imaging (HYDI) MRI.

METHODS

HYDI diffusion data were acquired in two groups of older and younger healthy participants recruited from the Indiana Alzheimer's Disease Research Center and community. These data were then transformed into super-resolution diffusion images before the hippocampal subfield analyses. We studied the correlation between the subjects' age and the structural connectivity involving the hippocampal subfields and the connectivity between the whole hippocampus and the cerebral cortex.

RESULTS

Structural integrity derived from the tractography streamlines between the hippocampal subfields was reduced in older than younger adults.

CONCLUSION

The findings offered a new promising framework, and they opened avenues for future studies to explore the relationship between the structural connectivity in the hippocampal area and different types of dementia.

The miEye: Bench-top super-resolution microscope with cost-effective equipment

Commercial super-resolution (SR) imaging systems require a high budget, while current more affordable open source microscopy systems lack modularity and sometimes are too complex or lack reliability. We present miEye - a cost-effective microscope designed for high-resolution wide-field fluorescence imaging. The build is constructed using a CNC milled aluminum microscope body and commercially available optomechanics, with open-source Python-based microscope control, data visualization, and analysis software integration. The data acquisition software works robustly with commonly used industrial-grade complementary metal oxide semiconductor (iCMOS) cameras, performs IR beam back-reflection-based automatic focus stabilization, and allows for laser control via an Arduino-based laser relay. The open-source nature of the design is aimed to facilitate adaptation by the community. The build can be constructed for a cost of roughly 50 k €. It contains SM-fiber and MM-fiber excitation paths that are easy to interchange and an adaptable emission path. Also, it ensures <5 nm/min stability of the sample on all axes, and allows achieving <30 nm lateral resolution for dSTORM and DNA-PAINT single-molecule localization microscopy (SMLM) experiments. Thus it serves as a cost-effective and adaptable addition to the open source microscopy community and potentially will allow high-quality SR imaging even for limited-budget research groups.

"Free super-resolution MRI by BRICKD slices"

BRICKD slices refers to shifting the field-of-view by fractional pixel increments between slices; half pixel shifts are analogous to the common brick wall layout. We demonstrate that compressed sensing reconstructions can harness the added information content of this approach and lead to improved performance over a simple stacked slices approach. The method is simple and could be easily implemented on a clinical imaging system.

Imaging nanoscale nuclear structures with expansion microscopy

Commonly applied super-resolution light microscopies have provided insight into subcellular processes at the nanoscale. However, imaging depth, speed, throughput and cost remain significant challenges, limiting the numbers of three-dimensional (3D) nanoscale processes that can be investigated and the number of laboratories able to undertake such analysis. Expansion microscopy (ExM) solves many of these limitations, but its application to imaging nuclear processes has been constrained by concerns of unequal nuclear expansion. Here, we demonstrate the conditions for isotropic expansion of the nucleus at a resolution equal to or better than 120–130 nm (pre-expansion). Using the DNA damage response proteins BRCA1, 53BP1 (also known as TP53BP1) and RAD51 as exemplars, we quantitatively describe the 3D nanoscale organisation of over 50,000 DNA damage response structures. We demonstrate the ability to assess chromatin-regulated events and show the simultaneous assessment of four elements. This study thus demonstrates how ExM can contribute to the investigation of nanoscale nuclear processes.

Summary: Expansion microscopy provides quantitative insight into the impact of chromatin modifiers on spatiotemporal organisation of the DNA repair proteins BRCA1, 53BP1 and RAD51 at a resolution of 65–70 nm.

Deep-learning-based 3D super-resolution MRI radiomics model: superior predictive performance in preoperative T-staging of rectal cancer

OBJECTIVES

To investigate the feasibility and efficacy of a deep-learning (DL)-based three-dimensional (3D) super-resolution (SR) MRI radiomics model for preoperative T-staging prediction in rectal cancer (RC).

METHODS

Seven hundred six eligible RC patients (T1/2 = 287, T3/4 = 419) were retrospectively enrolled in this study and chronologically allocated into a training cohort (n = 565) and a validation cohort (n = 141). We conducted a deep-transfer-learning network on high-resolution (HR) T2-weighted imaging (T2WI) to enhance the z-resolution of the images and acquired the preoperative SRT2WI. The radiomics models named model_HRT2 and model_SRT2 were respectively constructed with high-dimensional quantitative features extracted from manually segmented volume of interests of HRT2WI and SRT2WI through the Least Absolute Shrinkage and Selection Operator method. The performances of the models were evaluated by ROC, calibration, and decision curves.

RESULTS

Model_SRT2 outperformed model_HRT2 (AUC 0.869, sensitivity 71.1%, specificity 93.1%, and accuracy 83.3% vs. AUC 0.810, sensitivity 89.5%, specificity 70.1%, and accuracy 77.3%) in distinguishing T1/2 and T3/4 RC with significant difference (p < 0.05). Both radiomics models achieved higher AUCs than the expert radiologists (0.685, 95% confidence interval 0.595-0.775, p < 0.05). The calibration curves confirmed high goodness of fit, and the decision curve analysis revealed the clinical value.

CONCLUSIONS

Model_SRT2 yielded superior predictive performance in preoperative RC T-staging by comparison with model_HRT2 and expert radiologists' visual assessments.

KEY POINTS

• For the first time, DL-based 3D SR images were applied in radiomics analysis for clinical utility. • Compared with the visual assessment of expert radiologists and the conventional radiomics model based on HRT2WI, the SR radiomics model showed a more favorable capability in helping clinicians assess the invasion depth of RC preoperatively. • This is the largest radiomics study for T-staging prediction in RC.

Super-Resolution Reconstruction Based on BM3D and Compressed Sensing

In the various papers published in the field of super-resolution microscopy, denoising of raw images based on Block-matching and 3D filtering (BM3D) was rarely reported. BM3D for blocks of different sizes was studied. The denoising ability is related to block sizes. The larger the block is, the better the denoising effect is. When the block size is bigger than 40, the good denoising effect can be achieved. Denoising has great influence on the super-resolution reconstruction effect and the reconstruction time. Better super-resolution reconstruction and shorter reconstruction time can be achieved after denoising. Using compressed sensing, only 20 raw images are needed for super-resolution reconstruction. The temporal resolution is less than half a second. The spatial resolution is also greatly improved.

Dual U-Net residual networks for cardiac magnetic resonance images super-resolution

BACKGROUND AND OBJECTIVE

Heart disease is a vital disease that has threatened human health, and is the number one killer of human life. Moreover, with the added influence of recent health factors, its incidence rate keeps showing an upward trend. Today, cardiac magnetic resonance (CMR) imaging can provide a full range of structural and functional information for the heart, and has become an important tool for the diagnosis and treatment of heart disease. Therefore, improving the image resolution of CMR has an important medical value for the diagnosis and condition assessment of heart disease. At present, most single-image super-resolution (SISR) reconstruction methods have some serious problems, such as insufficient feature information mining, difficulty to determine the dependence of each channel of feature map, and reconstruction error when reconstructing high-resolution image.

METHODS

To solve these problems, we have proposed and implemented a dual U-Net residual network (DURN) for super-resolution of CMR images. Specifically, we first propose a U-Net residual network (URN) model, which is divided into the up-branch and the down-branch. The up-branch is composed of residual blocks and up-blocks to extract and upsample deep features; the down-branch is composed of residual blocks and down-blocks to extract and downsample deep features. Based on the URN model, we employ this a dual U-Net residual network (DURN) model, which combines the extracted deep features of the same position between the first URN and the second URN through residual connection. It can make full use of the features extracted by the first URN to extract deeper features of low-resolution images.

RESULTS

When the scale factors are 2, 3, and 4, our DURN can obtain 37.86 dB, 33.96 dB, and 31.65 dB on the Set5 dataset, which shows (i) a maximum improvement of 4.17 dB, 3.55 dB, and 3.22dB over the Bicubic algorithm, and (ii) a minimum improvement of 0.34 dB, 0.14 dB, and 0.11 dB over the LapSRN algorithm.

CONCLUSION

Comprehensive experimental study results on benchmark datasets demonstrate that our proposed DURN can not only achieve better performance for peak signal to noise ratio (PSNR) and structural similarity index (SSIM) values than other state-of-the-art SR image algorithms, but also reconstruct clearer super-resolution CMR images which have richer details, edges, and texture.

Single Molecule Tracking Nanoscopy Extended to Two Colors with MTT2col for the Analysis of Cell-Cell Interactions in Leukemia

Single molecule tracking (SMT) is a powerful technique to study molecular dynamics, and is particularly adapted to monitor the motion and interactions of cell membrane components. Assessing interactions among two molecular populations is classically performed by several approaches, including dual-color videomicroscopy, which allows monitoring of interactions through colocalization events. Other techniques, such as fluorescence recovery after photobleaching (FRAP), Förster resonance energy transfer (FRET), and fluorescence correlation spectroscopy (FCS), are also utilized to measure molecular dynamics. We developed MTT2col, a set of algorithmic tools extending multi-target tracing (MTT) to dual-color acquisition (https://github.com/arnauldserge1/MTT2col). In this protocol, we used MTT2col to monitor adhesion molecules at the contact between leukemic stem cells and stromal cells, a process involved in cancer resistance to chemotherapy and in relapse. Our dual-color single molecule protocol includes the following steps: (i) labeling molecules of interest with fluorescent probes, (ii) video-acquisition, (iii) analyses using our MTT2col in-house software, to obtain positions and trajectories, followed by (iv) detailed analyses of colocalization, distribution, and dynamic motion modes, according to the issues addressed. MTT2col is a robust and efficient SMT algorithm. Both MTT and MTT2col are open-source software that can be adapted and further developed for specific analyses. Graphical abstract.

Simultaneous Multicolor DNA-PAINT without Sequential Fluid Exchange Using Spectral Demixing

Several variants of multicolor single-molecule localization microscopy (SMLM) have been developed to resolve the spatial relationship of nanoscale structures in biological samples. The oligonucleotide-based SMLM approach "DNA-PAINT" robustly achieves nanometer localization precision and can be used to count binding sites within nanostructures. However, multicolor DNA-PAINT has primarily been realized by "Exchange-PAINT", which requires sequential exchange of the imaging solution and thus leads to extended acquisition times. To alleviate the need for fluid exchange and to speed up the acquisition of current multichannel DNA-PAINT, we here present a novel approach that combines DNA-PAINT with simultaneous multicolor acquisition using spectral demixing (SD). By using newly designed probes and a novel multichannel registration procedure, we achieve simultaneous multicolor SD-DNA-PAINT with minimal crosstalk. We demonstrate high localization precision (3-6 nm) and multicolor registration of dual- and triple-color SD-DNA-PAINT by resolving patterns on DNA origami nanostructures and cellular structures.

Deep Learning-Based 3D MRI Contrast-Enhanced Synthesis From A 2D Non-contrast T2Flair Sequence Deep Learning, MR images Synthesis

PURPOSE

Gadolinium-based contrast agents (GBCAs) have been successfully applied in magnetic resonance (MR) imaging to facilitate better lesion visualization. However, gadolinium deposition in the human brain raised widespread concerns recently. On the other hand, although high-resolution three-dimensional (3D) MR images are more desired for most existing medical image processing algorithms, their long scan duration and high acquiring costs make 2D MR images still much more common clinically. Therefore, developing alternative solutions for 3D contrast-enhanced MR images synthesis to replace GBCAs injection becomes an urgent requirement.

METHODS

This study proposed a deep learning framework that produces 3D isotropic full-contrast T2Flair images from 2D anisotropic non-contrast T2Flair image stacks. The super-resolution (SR) and contrast-enhanced (CE) synthesis tasks are completed in sequence by using an identical generative adversarial network (GAN) with the same techniques. To solve the problem that intra-modality datasets from different scanners have specific combinations of orientations, contrasts and resolutions, we conducted a region-based data augmentation technique on the fly during training to simulate various imaging protocols in the clinic. We further improved our network by introducing atrous spatial pyramid pooling, enhanced residual blocks and deep supervision for better quantitative and qualitative results.

RESULTS

Our proposed method achieved superior CE synthesized performance in quantitative metrics and perceptual evaluation. Detailedly, the PSNR, SSIM and AUC are 32.25 dB, 0.932 and 0.991 in the whole brain and 24.93 dB, 0.851 and 0.929 in tumor regions. The radiologists' evaluations confirmed that our proposed method has high confidence in the diagnosis. Analysis of the generalization ability showed that benefiting from the proposed data augmentation technique, our network can be applied to 'unseen' datasets with slight drops in quantitative and qualitative results.

CONCLUSION

Our work demonstrates the clinical potential of synthesizing diagnostic 3D isotropic CE brain MR images from a single 2D anisotropic non-contrast sequence. This article is protected by copyright. All rights reserved.

Disparity-constrained stereo endoscopic image super-resolution

PURPOSE

With the increasing usage of stereo cameras in computer-assisted surgery techniques, surgeons can benefit from better 3D context of the surgical site in minimally invasive operations. However, since stereo cameras are placed together at the confined endoscope tip, the size of lens and sensors is limited, resulting in low resolution of stereo endoscopic images. How to effectively exploit and utilize stereo information in stereo endoscopic super-resolution (SR) becomes a challenging problem.

METHODS

In this work, we propose a disparity-constrained stereo super-resolution network (DCSSRnet) to reconstruct images using a stereo image pair. In particular, a disparity constraint mechanism is incorporated into the generation of SR images in the deep neural network framework with effective feature extractors and atrous parallax attention modules.

RESULTS

Extensive experiments were conducted to evaluate the performance of proposed DCSSRnet on the da Vinci dataset and Medtronic dataset. The results on endoscopic image datasets demonstrate that the proposed approach produces a more effective improvement over current SR methods on both quantitative measurements. The ablation studies further verify the effectiveness of the components of the proposed framework.

CONCLUSION

The proposed DCSSRnet provides a promising solution on enhancing the spatial resolution of stereo endoscopic image pairs. Specifically, the disparity consistency of the stereo image pair provides informative supervision for image reconstruction. The proposed model can serve as a tool for improving the quality of stereo endoscopic images of endoscopic surgery systems.

Surgical biomicroscopy-guided intra-operative optical coherence tomography (iOCT) image super-resolution

Purpose

Intra-retinal delivery of novel sight-restoring therapies will require the precision of robotic systems accompanied by excellent visualisation of retinal layers. Intra-operative Optical Coherence Tomography (iOCT) provides cross-sectional retinal images in real time but at the cost of image quality that is insufficient for intra-retinal therapy delivery.This paper proposes a super-resolution methodology that improves iOCT image quality leveraging spatiotemporal consistency of incoming iOCT video streams.

Methods

To overcome the absence of ground truth high-resolution (HR) images, we first generate HR iOCT images by fusing spatially aligned iOCT video frames. Then, we automatically assess the quality of the HR images on key retinal layers using a deep semantic segmentation model. Finally, we use image-to-image translation models (Pix2Pix and CycleGAN) to enhance the quality of LR images via quality transfer from the estimated HR domain.

Results

Our proposed methodology generates iOCT images of improved quality according to both full-reference and no-reference metrics. A qualitative study with expert clinicians also confirms the improvement in the delineation of pertinent layers and in the reduction of artefacts. Furthermore, our approach outperforms conventional denoising filters and the learning-based state-of-the-art.

Conclusions

The results indicate that the learning-based methods using the estimated, through our pipeline, HR domain can be used to enhance the iOCT image quality. Therefore, the proposed method can computationally augment the capabilities of iOCT imaging helping this modality support the vitreoretinal surgical interventions of the future.

Virus morphology: Insights from super-resolution fluorescence microscopy

As epitomised by the COVID-19 pandemic, diseases caused by viruses are one of the greatest health and economic burdens to human society. Viruses are ‘nanostructures’, and their small size (typically less than 200 nm in diameter) can make it challenging to obtain images of their morphology and structure. Recent advances in fluorescence microscopy have given rise to super-resolution techniques, which have enabled the structure of viruses to be visualised directly at a resolution in the order of 20 nm. This mini-review discusses how recent state-of-the-art super-resolution imaging technologies are providing new nanoscale insights into virus structure.

Super-resolution Chromatin Visualization Using a Combined Method of Fluorescence In Situ Hybridization and Structured Illumination Microscopy in Solanum lycopersicum

Chromatin organization influences gene and transposon expression, and regulates various cellular processes. Higher order chromatin structure has been widely studied using genomic approaches and microscopy image analyses. Chromosome conformation capture and sequencing the junction of DNA fragments enables the study of both chromatin interaction and chromosome folding. However, certain cell types are embedded in other cell types which complicate the process of studying them using high-throughput genomic approaches. To overcome this limitation, high-resolution microscopy techniques are now available to investigate chromatin organization in single cells. In this chapter, we provide a detailed protocol to prepare chromosome spreading from tomato nuclei, to label genomic loci by fluorescence in situ hybridization, and to visualize these locations at high resolution with Structured Illumination microscopy.

Super-Resolution Imaging of Plasmodesmata Using 3D Structured Illumination Microscopy

Plasmodesmata (PD) have a diameter of around 30-50 nm which is well below the 200 nm limit of optical resolution, making analysis by light microscopy difficult and resolving internal structures of the PD such as the desmotubule impossible. Modern super-resolution methods such as 3D structured illumination microscopy (3D-SIM) can increase the lateral and axial resolution and work well on fixed, sectioned material. However, imaging in live plant cells requires careful optimization. Here we present a method to image PD using 3D-SIM in live BY2 cells.

Super-Resolution Radial Fluctuations (SRRF) Microscopy

Super-resolution Radial Fluctuations (SRRF) imaging is a computational approach to fixed and live-cell super-resolution microscopy that is highly accessible to life science researchers since it uses common microscopes and open-source software plugins for ImageJ. This allows users to generate super-resolution images using the same equipment, fluorophores, fluorescent proteins and methods they routinely employ for their studies without specialized sample preparations or reagents. Here, we discuss a step-by-step workflow for acquiring and analyzing images using the NanoJ-SRRF software developed by the Ricardo Henriques group, with a focus on imaging chromatin. Increased accessibility of affordable super-resolution imaging techniques is an important step in extending the reach of this revolution in cellular imaging to a greater number of laboratories.

A Next Generation of Advances in Chromosome Architecture

New insight into the architecture of chromosomes, their molecular composition, structure and spatial location, and time-resolved features, has grown enormously through developments of a range of pioneering interdisciplinary approaches that lie at the interface of the life and physical sciences. These involve several state-of-the-art "physics of life" tools that are both experimental and theoretical, used in conjunction with molecular biology methods which enable investigation of chromosome structure and function in vitro, in vivo, and even in silico. In particular, a move towards far greater quantitation has enabled transformative leaps in our understanding. These have involved valuable improvements to the spatial and temporal resolution of quantitative measurements, such as in vivo super-resolved light microscopy and single-molecule biophysics methods, which facilitate probing of dynamic chromosome processes hitherto impossible. Similarly, there have been important advances in the theoretical biophysics approaches which have enabled advances in predictive modeling to generate new understanding of the modes of operation of chromosomes across all domains of life. Here, I discuss these advances, and review the current state of our knowledge of chromosome architecture and speculation where future advances may lead.

Adversarial example defense based on image reconstruction

The rapid development of deep neural networks (DNN) has promoted the widespread application of image recognition, natural language processing, and autonomous driving. However, DNN is vulnerable to adversarial examples, such as an input sample with imperceptible perturbation which can easily invalidate the DNN and even deliberately modify the classification results. Therefore, this article proposes a preprocessing defense framework based on image compression reconstruction to achieve adversarial example defense. Firstly, the defense framework performs pixel depth compression on the input image based on the sensitivity of the adversarial example to eliminate adversarial perturbations. Secondly, we use the super-resolution image reconstruction network to restore the image quality and then map the adversarial example to the clean image. Therefore, there is no need to modify the network structure of the classifier model, and it can be easily combined with other defense methods. Finally, we evaluate the algorithm with MNIST, Fashion-MNIST, and CIFAR-10 datasets; the experimental results show that our approach outperforms current techniques in the task of defending against adversarial example attacks.

Nanoscale characterization of drug-induced microtubule filament dysfunction using super-resolution microscopy

BACKGROUND

The integrity of microtubule filament networks is essential for the roles in diverse cellular functions, and disruption of its structure or dynamics has been explored as a therapeutic approach to tackle diseases such as cancer. Microtubule-interacting drugs, sometimes referred to as antimitotics, are used in cancer therapy to target and disrupt microtubules. However, due to associated side effects on healthy cells, there is a need to develop safer drug regimens that still retain clinical efficacy. Currently, many questions remain open regarding the extent of effects on cellular physiology of microtubule-interacting drugs at clinically relevant and low doses. Here, we use super-resolution microscopies (single-molecule localization and optical fluctuation based) to reveal the initial microtubule dysfunctions caused by nanomolar concentrations of colcemid.

RESULTS

We identify previously undetected microtubule (MT) damage caused by clinically relevant doses of colcemid. Short exposure to 30-80 nM colcemid results in aberrant microtubule curvature, with a trend of increased curvature associated to increased doses, and curvatures greater than 2 rad/μm, a value associated with MT breakage. Microtubule fragmentation was detected upon treatment with ≥ 100 nM colcemid. Remarkably, lower doses (< 20 nM after 5 h) led to subtle but significant microtubule architecture remodelling characterized by increased curvature and suppression of microtubule dynamics.

CONCLUSIONS

Our results support the emerging hypothesis that microtubule-interacting drugs induce non-mitotic effects in cells, and establish a multi-modal imaging assay for detecting and measuring nanoscale microtubule dysfunction. The sub-diffraction visualization of these less severe precursor perturbations compared to the established antimitotic effects of microtubule-interacting drugs offers potential for improved understanding and design of anticancer agents.

Dilated projection correction network based on autoencoder for hyperspectral image super-resolution

This paper focuses on improving the spatial resolution of the hyperspectral image (HSI) by taking the prior information into consideration. In recent years, single HSI super-resolution methods based on deep learning have achieved good performance. However, most of them only simply apply general image super-resolution deep networks to hyperspectral data, thus ignoring some specific characteristics of hyperspectral data itself. In order to make full use of spectral information of the HSI, we transform the HSI SR problem from the image domain into the abundance domain by the dilated projection correction network with an autoencoder, termed as aeDPCN. In particular, we first encode the low-resolution HSI to abundance representation and preserve the spectral information in the decoder network, which could largely reduce the computational complexity. Then, to enhance the spatial resolution of the abundance embedding, we super-resolve the embedding in a coarse-to-fine manner by the dilated projection correction network where the back-projection strategy is introduced to further eliminate spectral distortion. Finally, the predictive images are derived by the same decoder, which increases the stability of our method, even at a large upscaling factor. Extensive experiments on real hyperspectral image scenes demonstrate the superiority of our method over the state-of-the-art, in terms of accuracy and efficiency.

Fine-tuning deep learning model parameters for improved super-resolution of dynamic MRI with prior-knowledge

Dynamic imaging is a beneficial tool for interventions to assess physiological changes. Nonetheless during dynamic MRI, while achieving a high temporal resolution, the spatial resolution is compromised. To overcome this spatio-temporal trade-off, this research presents a super-resolution (SR) MRI reconstruction with prior knowledge based fine-tuning to maximise spatial information while reducing the required scan-time for dynamic MRIs. A U-Net based network with perceptual loss is trained on a benchmark dataset and fine-tuned using one subject-specific static high resolution MRI as prior knowledge to obtain high resolution dynamic images during the inference stage. 3D dynamic data for three subjects were acquired with different parameters to test the generalisation capabilities of the network. The method was tested for different levels of in-plane undersampling for dynamic MRI. The reconstructed dynamic SR results after fine-tuning showed higher similarity with the high resolution ground-truth, while quantitatively achieving statistically significant improvement. The average SSIM of the lowest resolution experimented during this research (6.25% of the k-space) before and after fine-tuning were 0.939 ± 0.008 and 0.957 ± 0.006 respectively. This could theoretically result in an acceleration factor of 16, which can potentially be acquired in less than half a second. The proposed approach shows that the super-resolution MRI reconstruction with prior-information can alleviate the spatio-temporal trade-off in dynamic MRI, even for high acceleration factors.

Super-resolution of Pneumocystis carinii pneumonia CT via self-attention GAN

BACKGROUND AND OBJECTIVE

Computed tomography (CT) examination plays an important role in screening suspected and confirmed patients in pneumocystis carinii pneumonia (PCP), and the efficient acquisition of high-quality medical CT images is essential for the clinical application of computer-aided diagnosis technology. Therefore, improving the resolution of CT images of pneumonia is a very important task.

METHODS

Aiming at the problem of how to recover the texture details of the reconstructed PCP CT super-resolution image, we propose the image super-resolution reconstruction model based on self-attention generation adversarial network (SAGAN). In the SAGAN algorithm, a generator based on self-attention mechanism and residual module is used to transform a low-resolution image into a super-resolution image. A discriminator based on depth convolution network tries to distinguish the difference between the reconstructed super-resolution image and the real super-resolution image. In terms of loss function construction, on the one hand, the Charbonnier content loss function is used to improve the accuracy of image reconstruction, and on the other hand, the feature value before activation of the pre-trained VGGNet is used to calculate the perceptual loss to achieve accurate texture detail reconstruction of super-resolution images.

RESULTS

Experimental results show that our SAGAN algorithm is superior to other state-of-the-art algorithms in both peak signal-to-noise ratio (PSNR) and structural similarity score (SSIM). Specifically, our SAGAN method can obtain 31.94 dB which is 1.53 dB better than SRGAN on Set5 dataset for 4 enlargements.

CONCLUSION

Our SAGAN method can reconstruct more realistic PCP CT images with clear texture, which can help experts diagnose the condition of PCP.

A Magnified Adaptive Feature Pyramid Network for automatic microaneurysms detection

Diabetic retinopathy (DR), as an important complication of diabetes, is the primary cause of blindness in adults. Automatic DR detection poses a challenge which is crucial for early DR screening. Currently, the vast majority of DR is diagnosed through fundus images, where the microaneurysm (MA) has been widely used as the most distinguishable marker. Research works on automatic DR detection have traditionally utilized manually designed operators, while a few recent researchers have explored deep learning techniques for this topic. But due to issues such as the extremely small size of microaneurysms, low resolution of fundus pictures, and insufficient imaging depth, the DR detection problem is quite challenging and remains unsolved. To address these issues, this research proposes a new deep learning model (Magnified Adaptive Feature Pyramid Network, MAFP-Net) for DR detection, which conducts super-resolution on low quality fundus images and integrates an improved feature pyramid structure while utilizing a standard two-stage detection network as the backbone. Our proposed detection model needs no pre-segmented patches to train the CNN network. When tested on the E-ophtha-MA dataset, the sensitivity value of our method reached as high as 83.5% at false positives per image (FPI) of 8 and the F1 value achieved 0.676, exceeding all those of the state-of-the-art algorithms as well as the human performance of experienced physicians. Similar results were achieved on another public dataset of IDRiD.

MRI Super-Resolution Through Generative Degradation Learning

Spatial resolution plays a critically important role in MRI for the precise delineation of the imaged tissues. Unfortunately, acquisitions with high spatial resolution require increased imaging time, which increases the potential of subject motion, and suffers from reduced signal-to-noise ratio (SNR). Super-resolution reconstruction (SRR) has recently emerged as a technique that allows for a trade-off between high spatial resolution, high SNR, and short scan duration. Deconvolution-based SRR has recently received significant interest due to the convenience of using the image space. The most critical factor to succeed in deconvolution is the accuracy of the estimated blur kernels that characterize how the image was degraded in the acquisition process. Current methods use handcrafted filters, such as Gaussian filters, to approximate the blur kernels, and have achieved promising SRR results. As the image degradation is complex and varies with different sequences and scanners, handcrafted filters, unfortunately, do not necessarily ensure the success of the deconvolution. We sought to develop a technique that enables accurately estimating blur kernels from the image data itself. We designed a deep architecture that utilizes an adversarial scheme with a generative neural network against its degradation counterparts. This design allows for the SRR tailored to an individual subject, as the training requires the scan-specific data only, i.e., it does not require auxiliary datasets of high-quality images, which are practically challenging to obtain. With this technique, we achieved high-quality brain MRI at an isotropic resolution of 0.125 cubic mm with six minutes of imaging time. Extensive experiments on both simulated low-resolution data and clinical data acquired from ten pediatric patients demonstrated that our approach achieved superior SRR results as compared to state-of-the-art deconvolution-based methods, while in parallel, at substantially reduced imaging time in comparison to direct high-resolution acquisitions.

A Generative Adversarial Network technique for high-quality super-resolution reconstruction of cardiac magnetic resonance images

PURPOSE

In this paper, we proposed a Denoising Super-resolution Generative Adversarial Network (DnSRGAN) method for high-quality super-resolution reconstruction of noisy cardiac magnetic resonance (CMR) images.

METHODS

The proposed method is based on feed-forward denoising convolutional neural network (DnCNN) and SRGAN architecture. Firstly, we used a feed-forward denoising neural network to pre-denoise the CMR image to ensure that the input is a clean image. Secondly, we use the gradient penalty (GP) method to solve the problem of the discriminator gradient disappearing, which improves the convergence speed of the model. Finally, a new loss function is added to the original SRGAN loss function to monitor GAN gradient descent to achieve more stable and efficient model training, thereby providing higher perceptual quality for the super-resolution of CMR images.

RESULTS

We divided the tested cardiac images into 3 groups, each group of 25 images. Then, we calculated the Peak Signal to Noise Ratio (PSNR) /Structural Similarity (SSIM) between Ground Truth (GT) and the images generated by super-resolution, used them to evaluate our model. We compared with the current widely used method: Bicubic ESRGAN and SRGAN, our method has better reconstruction quality and higher PSNR/SSIM score.

CONCLUSION

We used DnCNN to denoise the CMR image, and then using the improved SRGAN to perform super-resolution reconstruction of the denoised image, we can solve the problem of high noise and artifacts that cause the cardiac image to be reconstructed incorrectly during super-resolution.

A diagnostic information based framework for super-resolution and quality assessment of retinal OCT images

High-resolution (HR) retinal optical coherence tomography (OCT) images are preferred by the ophthalmologists to diagnose retinal diseases. These images can be obtained by dense scanning of the target retinal region during acquisition. However, a dense scanning increases the image acquisition time and introduces motion artefacts, which corrupt diagnostic information. Therefore, researchers have a growing interest in developing image processing techniques to recover HR images from low-resolution (LR) OCT images. In this paper, we present an automated super-resolution (SR) scheme using diagnostic information weighted sparse representation framework to reconstruct HR images from LR OCT images. The proposed method performs fast and reliable reconstruction of the LR images. We also propose a 2D- variational mode decomposition (VMD) based OCT diagnostic distortion measure (Q_OCT) to quantify diagnostic distortion in the reconstructed OCT images. The SR method is evaluated on clinical grade OCT images with the proposed diagnostic distortion measure along with the conventional non-diagnostic measures like the contrast to noise ratio (CNR), the equivalent number of looks (ENL) and the peak signal to noise ratio (PSNR). The results show an average CNR of 4.07, ENL of 58.96 and PSNR of 27.72 dB. An average score of 1.53 is obtained using the proposed diagnostic distortion measure. Experimental results quantify that the proposed Q_OCT metric can effectively capture diagnostic distortion.

A terahertz time-domain super-resolution imaging method using a local-pixel graph neural network for biological products

The low image acquisition speed of terahertz (THz) time-domain imaging systems limits their application in biological products analysis. In the current study, a local pixel graph neural network was built for THz time-domain imaging super-resolution. The method could be applied to the analysis of any heterogeneous biological products as it only required a small number of sample images for training and particularly it focused on THz feature frequencies. The graph network applied the Fourier transform to graphs extracted from low-resolution (LR) images bringing an invariance of rotation and flip for local pixels, and the network then learnt the relationship between the state of graphs and the corresponding pixels to be reconstructed. With wood cores and seeds as examples, the images of these samples were captured by a THz time-domain imaging system for training and analysed by the method, achieving the root mean square error (RMSE) of pixels of 0.0957 and 0.1061 for the wood core and seed images, respectively. In addition, the reconstructed high-resolution (HR) images, LR images and true HR images at several feature frequencies were also compared in the current study. Results indicated that the method could not only reconstruct the spatial details and the useful signals from high noise signals at high feature frequencies but could also operate super-resolution in both spatial and spectral aspects.

Artificial Intelligence-Based Image Enhancement in PET Imaging: Noise Reduction and Resolution Enhancement

High noise and low spatial resolution are two key confounding factors that limit the qualitative and quantitative accuracy of PET images. Artificial intelligence models for image denoising and deblurring are becoming increasingly popular for the postreconstruction enhancement of PET images. We present a detailed review of recent efforts for artificial intelligence-based PET image enhancement with a focus on network architectures, data types, loss functions, and evaluation metrics. We also highlight emerging areas in this field that are quickly gaining popularity, identify barriers to large-scale adoption of artificial intelligence models for PET image enhancement, and discuss future directions.

3D Isotropic Super-resolution Prostate MRI Using Generative Adversarial Networks and Unpaired Multiplane Slices

We developed a deep learning-based super-resolution model for prostate MRI. 2D T2-weighted turbo spin echo (T2w-TSE) images are the core anatomical sequences in a multiparametric MRI (mpMRI) protocol. These images have coarse through-plane resolution, are non-isotropic, and have long acquisition times (approximately 10-15 min). The model we developed aims to preserve high-frequency details that are normally lost after 3D reconstruction. We propose a novel framework for generating isotropic volumes using generative adversarial networks (GAN) from anisotropic 2D T2w-TSE and single-shot fast spin echo (ssFSE) images. The CycleGAN model used in this study allows the unpaired dataset mapping to reconstruct super-resolution (SR) volumes. Fivefold cross-validation was performed. The improvements from patch-to-volume reconstruction (PVR) to SR are 80.17%, 63.77%, and 186% for perceptual index (PI), RMSE, and SSIM, respectively; the improvements from slice-to-volume reconstruction (SVR) to SR are 72.41%, 17.44%, and 7.5% for PI, RMSE, and SSIM, respectively. Five ssFSE cases were used to test for generalizability; the perceptual quality of SR images surpasses the in-plane ssFSE images by 37.5%, with 3.26% improvement in SSIM and a higher RMSE by 7.92%. SR images were quantitatively assessed with radiologist Likert scores. Our isotropic SR volumes are able to reproduce high-frequency detail, maintaining comparable image quality to in-plane TSE images in all planes without sacrificing perceptual accuracy. The SR reconstruction networks were also successfully applied to the ssFSE images, demonstrating that high-quality isotropic volume achieved from ultra-fast acquisition is feasible.

FA-GAN: Fused attentive generative adversarial networks for MRI image super-resolution

High-resolution magnetic resonance images can provide fine-grained anatomical information, but acquiring such data requires a long scanning time. In this paper, a framework called the Fused Attentive Generative Adversarial Networks(FA-GAN) is proposed to generate the super- resolution MR image from low-resolution magnetic resonance images, which can reduce the scanning time effectively but with high resolution MR images. In the framework of the FA-GAN, the local fusion feature block, consisting of different three-pass networks by using different convolution kernels, is proposed to extract image features at different scales. And the global feature fusion module, including the channel attention module, the self-attention module, and the fusion operation, is designed to enhance the important features of the MR image. Moreover, the spectral normalization process is introduced to make the discriminator network stable. 40 sets of 3D magnetic resonance images (each set of images contains 256 slices) are used to train the network, and 10 sets of images are used to test the proposed method. The experimental results show that the PSNR and SSIM values of the super-resolution magnetic resonance image generated by the proposed FA-GAN method are higher than the state-of-the-art reconstruction methods.

Residual dense network for medical magnetic resonance images super-resolution

BACKGROUND AND OBJECTIVE

High-resolution magnetic resonance images (MRI) help experts to localize lesions and diagnose diseases, but it is difficult to obtain high-resolution MRI. Furthermore, image super-resolution technology based on deep learning can effectively improve image resolution.

METHODS

In this work, we propose a medical magnetic resonance (MR) image super-resolution reconstruction method based on residual dense network (MRDN). Firstly, we input the convolutional features of the shallow layer into the residual dense block to obtain global and local features. Secondly, each layer in the residual dense block is directly connected to the previous layer to achieve reuse of features. Finally, we use sub-pixel convolution layer for upsampling and super-resolution reconstruction to get a clear high-resolution image.

RESULTS

For the 2 ×, 3 ×, and 4 × enlargement, we propose the MRDN method shows the superiority over the state-of-the-art methods on the Set5, Set14, and Urban100 benchmark datasets, extensive benchmark experiment and analysis show that the superiority of our MRDN algorithm in terms of the peak signal-to-noise ratio (PSNR) and structural similarity index indicators (SSIM).

CONCLUSION

Quantitative experiments are conducted on three public datasets: Set5, Set14 and Urban10, evaluate with commonly used evaluation metrics, and the experimental results show that the method in this paper is more effective. In addition, we reconstruct the public MR datasets, and the reconstructed high-resolution MR image has a clear structure and rich texture details.

Gradual back-projection residual attention network for magnetic resonance image super-resolution

BACKGROUND AND OBJECTIVE

Magnetic Resonance Image (MRI) analysis can provide anatomical examination of internal organs, which is helpful for diagnosis of the disease. Aiming at the problems of insufficient feature information mining in the process of MRI super-resolution (SR) reconstruction, the difficulty of determining the interdependence between the channels of the feature map, and the reconstruction error when reconstructing high-resolution (HR) images, we propose a SR method to solve these problems.

METHODS

In this work, we propose a gradual back-projection residual attention network for MRI super-resolution (GRAN), which outperforms most of the state-of-the-art methods. Firstly, we use the gradual upsampling method to gradually scale the low-resolution (LR) image to a given magnification to alleviate the high-frequency information loss caused by the upsampling process. Secondly, we merge the idea of iterative back-projection at each stage of gradual upsampling, learn the mapping relationship between HR and LR feature maps and reduce the noise introduced during the upsampling process. Finally, we use the attention mechanism to dynamically allocate attention resources to the feature maps generated at different stages of the gradual back-projection network, so that the network model can learn the interdependence between each feature map.

RESULTS

For the 2 × and 4 × enlargement, the proposed GRAN method shows the superiority over the state-of-the-art methods on the Set5, Set14, and Urban100 benchmark datasets, extensive benchmark experiment and analysis show that the superiority of the GRAN algorithm in terms of peak signal-to-noise ratio and structural similarity index indicators.

CONCLUSION

The MRI results reconstructed by gradual back-projection residual attention network on the public dataset IDI have good image sharpness, rich texture details and good visual experience. In addition, the reconstructed image is the closest to the real image, enabling the medical expert to see the biological tissue structure and its early pathological changes more clearly, providing assistance and support to the medical expert in the diagnosis and treatment of the disease.

Progressive back-projection network for COVID-CT super-resolution

BACKGROUND AND OBJECTIVE

Recently, the COVID-19 epidemic has become more and more serious around the world, how to improve the image resolution of COVID-CT is a very important task. The network based on progressive upsampling for COVID-CT super-resolution increases the reconstruction error. This paper proposes a progressive back-projection network (PBPN) for COVID-CT super-resolution to solve this problem.

METHODS

In this paper, we propose a progressive back-projection network (PBPN) for COVID-CT super-resolution. PBPN is divided into two stages, and each stage consists of back-projection, deep feature extraction and upscaling. We design an up-projection and down-projection residual module to minimize the reconstruction error and construct a residual attention module to extract deep features. In each stage, firstly, PBPN performs back-projection to extract shallow features by two up-projection and down-projection residual modules; then, PBPN extracts deep features from the shallow features by two residual attention modules; finally, PBPN upsamples the deep features through sub-pixel convolution.

RESULTS

The proposed method achieves the improvements of about 0.14~0.47 dB/0.0012~0.0060 for × 2 scale factor, 0.02~0.08 dB/0.0024~0.0059 for × 3 scale factor, and 0.08~0.41 dB/ 0.0040~0.0147 for × 4 scale factor than state-of-the-art methods (Bicubic, SRCNN, FSRCNN, VDSR, LapSRN, DRCN and DSRN) in terms of PSNR/SSIM on benchmark datasets.

CONCLUSIONS

The proposed mehtod obtains better performance for COVID-CT super-resolution and reconstructs high-quality high-resolution COVID-CT images that contain more details and edges.

Instant multicolor super-resolution microscopy with deep convolutional neural network

Multicolor super-resolution (SR) microscopy plays a critical role in cell biology research and can visualize the interactions between different organelles and the cytoskeleton within a single cell. However, more color channels bring about a heavier budget for imaging and sample preparation, and the use of fluorescent dyes of higher emission wavelengths leads to a worse spatial resolution. Recently, deep convolutional neural networks (CNNs) have shown a compelling capability in cell segmentation, super-resolution reconstruction, image restoration, and many other aspects. Taking advantage of CNN's strong representational ability, we devised a deep CNN-based instant multicolor super-resolution imaging method termed IMC-SR and demonstrated that it could be used to separate different biological components labeled with the same fluorophore, and generate multicolor images from a single super-resolution image in silico. By IMC-SR, we achieved fast three-color live-cell super-resolution imaging with ~100 nm resolution over a long temporal duration, revealing the complicated interactions between multiple organelles and the cytoskeleton in a single COS-7 cell.

A Novel Method to Map Small RNAs with High Resolution

Analyzing cellular structures and the relative location of molecules is essential for addressing biological questions. Super-resolution microscopy techniques that bypass the light diffraction limit have become increasingly popular to study cellular molecule dynamics in situ. However, the application of super-resolution imaging techniques to detect small RNAs (sRNAs) is limited by the choice of proper fluorophores, autofluorescence of samples, and failure to multiplex. Here, we describe an sRNA-PAINT protocol for the detection of sRNAs at nanometer resolution. The method combines the specificity of locked nucleic acid probes and the low background, precise quantitation, and multiplexable characteristics of DNA Point Accumulation for Imaging in Nanoscale Topography (DNA-PAINT). Using this method, we successfully located sRNA targets that are important for development in maize anthers at sub-20 nm resolution and quantitated their exact copy numbers. Graphic abstract: Multiplexed sRNA-PAINT. Multiple Vetting and Analysis of RNA for In Situ Hybridization (VARNISH) probes with different docking strands (i.e., a, b, …) will be hybridized to samples. The first probe will be imaged with the a* imager. The a* imager will be washed off with buffer C, and then the sample will be imaged with b* imager. The wash and image steps can be repeated sequentially for multiplexing.

Joint super-resolution and synthesis of 1 mm isotropic MP-RAGE volumes from clinical MRI exams with scans of different orientation, resolution and contrast

Most existing algorithms for automatic 3D morphometry of human brain MRI scans are designed for data with near-isotropic voxels at approximately 1 mm resolution, and frequently have contrast constraints as well-typically requiring T1-weighted images (e.g., MP-RAGE scans). This limitation prevents the analysis of millions of MRI scans acquired with large inter-slice spacing in clinical settings every year. In turn, the inability to quantitatively analyze these scans hinders the adoption of quantitative neuro imaging in healthcare, and also precludes research studies that could attain huge sample sizes and hence greatly improve our understanding of the human brain. Recent advances in convolutional neural networks (CNNs) are producing outstanding results in super-resolution and contrast synthesis of MRI. However, these approaches are very sensitive to the specific combination of contrast, resolution and orientation of the input images, and thus do not generalize to diverse clinical acquisition protocols - even within sites. In this article, we present SynthSR, a method to train a CNN that receives one or more scans with spaced slices, acquired with different contrast, resolution and orientation, and produces an isotropic scan of canonical contrast (typically a 1 mm MP-RAGE). The presented method does not require any preprocessing, beyond rigid coregistration of the input scans. Crucially, SynthSR trains on synthetic input images generated from 3D segmentations, and can thus be used to train CNNs for any combination of contrasts, resolutions and orientations without high-resolution real images of the input contrasts. We test the images generated with SynthSR in an array of common downstream analyses, and show that they can be reliably used for subcortical segmentation and volumetry, image registration (e.g., for tensor-based morphometry), and, if some image quality requirements are met, even cortical thickness morphometry. The source code is publicly available at https://github.com/BBillot/SynthSR.