Super-resolution microscopies, technological breakthrough to decipher mitochondrial structure and dynamic

Mitochondria are complex organelles with an outer membrane enveloping a second inner membrane that creates a vast matrix space partitioned by pockets or cristae that join the peripheral inner membrane with several thin junctions. Several micrometres long, mitochondria are generally close to 300 nm in diameter, with membrane layers separated by a few tens of nanometres. Ultrastructural data from electron microscopy revealed the structure of these mitochondria, while conventional optical microscopy revealed their extraordinary dynamics through fusion, fission, and migration processes but its limited resolution power restricted the possibility to go further. By overcoming the limits of light diffraction, Super-Resolution Microscopy (SRM) now offers the potential to establish the links between the ultrastructure and remodelling of mitochondrial membranes, leading to major advances in our understanding of mitochondria's structure-function. Here we review the contributions of SRM imaging to our understanding of the relationship between mitochondrial structure and function. What are the hopes for these new imaging approaches which are particularly important for mitochondrial pathologies?

Clinical Super-Resolution Computed Tomography of Bone Microstructure: Application in Musculoskeletal and Dental Imaging

PURPOSE

Clinical cone-beam computed tomography (CBCT) devices are limited to imaging features of half a millimeter in size and cannot quantify the tissue microstructure. We demonstrate a robust deep-learning method for enhancing clinical CT images, only requiring a limited set of easy-to-acquire training data.

METHODS

Knee tissue from five cadavers and six total knee replacement patients, and 14 teeth from eight patients were scanned using laboratory CT as training data for the developed super-resolution (SR) technique. The method was benchmarked against ex vivo test set, 52 osteochondral samples are imaged with clinical and laboratory CT. A quality assurance phantom was imaged with clinical CT to quantify the technical image quality. To visually assess the clinical image quality, musculoskeletal and maxillofacial CBCT studies were enhanced with SR and contrasted to interpolated images. A dental radiologist and surgeon reviewed the maxillofacial images.

RESULTS

The SR models predicted the bone morphological parameters on the ex vivo test set more accurately than conventional image processing. The phantom analysis confirmed higher spatial resolution on the SR images than interpolation, but image grayscales were modified. Musculoskeletal and maxillofacial CBCT images showed more details on SR than interpolation; however, artifacts were observed near the crown of the teeth. The readers assessed mediocre overall scores for both SR and interpolation. The source code and pretrained networks are publicly available.

CONCLUSION

Model training with laboratory modalities could push the resolution limit beyond state-of-the-art clinical musculoskeletal and dental CBCT. A larger maxillofacial training dataset is recommended for dental applications.

Super-resolution techniques for biomedical applications and challenges

Super-resolution (SR) techniques have revolutionized the field of biomedical applications by detailing the structures at resolutions beyond the limits of imaging or measuring tools. These techniques have been applied in various biomedical applications, including microscopy, magnetic resonance imaging (MRI), computed tomography (CT), X-ray, electroencephalogram (EEG), ultrasound, etc. SR methods are categorized into two main types: traditional non-learning-based methods and modern learning-based approaches. In both applications, SR methodologies have been effectively utilized on biomedical images, enhancing the visualization of complex biological structures. Additionally, these methods have been employed on biomedical data, leading to improvements in computational precision and efficiency for biomedical simulations. The use of SR techniques has resulted in more detailed and accurate analyses in diagnostics and research, essential for early disease detection and treatment planning. However, challenges such as computational demands, data interpretation complexities, and the lack of unified high-quality data persist. The article emphasizes these issues, underscoring the need for ongoing development in SR technologies to further improve biomedical research and patient care outcomes.

3D-MRI super-resolution reconstruction using multi-modality based on multi-resolution CNN

BACKGROUND AND OBJECTIVE

High-resolution (HR) MR images provide rich structural detail to assist physicians in clinical diagnosis and treatment plan. However, it is arduous to acquire HR MRI due to equipment limitations, scanning time or patient comfort. Instead, HR MRI could be obtained through a number of computer assisted post-processing methods that have proven to be effective and reliable. This paper aims to develop a convolutional neural network (CNN) based super-resolution reconstruction framework for low-resolution (LR) T2w images.

METHOD

In this paper, we propose a novel multi-modal HR MRI generation framework based on deep learning techniques. Specifically, we construct a CNN based on multi-resolution analysis to learn an end-to-end mapping between LR T2w and HR T2w, where HR T1w is fed into the network to offer detailed a priori information to help generate HR T2w. Furthermore, a low-frequency filtering module is introduced to filter out the interference from HR-T1w during high-frequency information extraction. Based on the idea of multi-resolution analysis, detailed features extracted from HR T1w and LR T2w are fused at two scales in the network and then HR T2w is reconstructed by upsampling and dense connectivity module.

RESULTS

Extensive quantitative and qualitative evaluations demonstrate that the proposed method enhances the recovered HR T2w details and outperforms other state-of-the-art methods. In addition, the experimental results also suggest that our network has a lightweight structure and favorable generalization performance.

CONCLUSION

The results show that the proposed method is capable of reconstructing HR T2w with higher accuracy. Meanwhile, the super-resolution reconstruction results on other dataset illustrate the excellent generalization ability of the method.

Improving radiomic modeling for the identification of symptomatic carotid atherosclerotic plaques using deep learning-based 3D super-resolution CT angiography

Rationale and objectives

Radiomic models based on normal-resolution (NR) computed tomography angiography (CTA) images can fail to distinguish between symptomatic and asymptomatic carotid atherosclerotic plaques. This study aimed to explore the effectiveness of a deep learning-based three-dimensional super-resolution (SR) CTA radiomic model for improved identification of symptomatic carotid atherosclerotic plaques.

Materials and methods

A total of 193 patients with carotid atherosclerotic plaques were retrospectively enrolled and allocated into either a symptomatic (n = 123) or an asymptomatic (n = 70) groups. SR CTA images were derived from NR CTA images using deep learning-based three-dimensional SR technology. Handcrafted radiomic features were extracted from both the SR and NR CTA images and three risk models were developed based on manually measured quantitative CTA characteristics and NR and SR radiomic features. Model performances were assessed via receiver operating characteristic, calibration, and decision curve analyses.

Results

The SR model exhibited the optimal performance (area under the curve [AUC] 0.820, accuracy 0.802, sensitivity 0.854, F1 score 0.847) in the testing cohort, outperforming the other two models. The calibration curve analyses and Hosmer-Lemeshow test demonstrated that the SR model exhibited the best goodness of fit, and decision curve analysis revealed that SR model had the highest clinical value and potential patient benefits.

Conclusions

Deep learning-based three-dimensional SR technology could improve the CTA-based radiomic models in identifying symptomatic carotid plaques, potentially providing more accurate and valuable information to guide clinical decision-making to reduce the risk of ischemic stroke.

Nonlinear super-resolution signal processing allows intracellular tracking of calcium dynamics

OBJECTIVE

Traditional quantification of fluorescence signals, such as ∆F/F, relies on ratiometric measures that necessitate a baseline for compar- ison, limiting their applicability in dynamic analyses. Our goal here is to develop a baseline-independent method for analyzing fluorescence data that fully exploits temporal dynamics to introduce a novel approach for dynami- cal super-resolution analysis, including in subcellular resolution. Approach: We introduce ARES (Autoregressive RESiduals), a novel method that leverages the temporal aspect of fluorescence signals. By focusing on the quantification of residuals following linear autoregression, ARES obviates the need for a predefined baseline, enabling a more nuanced analysis of signal dynamics. Main Result: We delineate the foundational attributes of ARES, illustrat- ing its capability to enhance both spatial and temporal resolution of calcium fluorescence activity beyond the conventional ratiometric measure (∆F/F). Additionally, we demonstrate ARES's utility in elucidating intracellular cal- cium dynamics through the detailed observation of calcium wave propagation within a dendrite. Significance: ARES stands out as a robust and precise tool for the quan- tification of fluorescence signals, adept at analyzing both spontaneous and evoked calcium dynamics. Its ability to facilitate the subcellular localiza- tion of calcium signals and the spatiotemporal tracking of calcium dynam- ics-where traditional ratiometric measures falter-underscores its potential to revolutionize baseline-independent analyses in the field.

MRI super-resolution using similarity distance and multi-scale receptive field based feature fusion GAN and pre-trained slice interpolation network

Challenges arise in achieving high-resolution Magnetic Resonance Imaging (MRI) to improve disease diagnosis accuracy due to limitations in hardware, patient discomfort, long acquisition times, and high costs. While Convolutional Neural Networks (CNNs) have shown promising results in MRI super-resolution, they often don't look into the structural similarity and prior information available in consecutive MRI slices. By leveraging information from sequential slices, more robust features can be obtained, potentially leading to higher-quality MRI slices. We propose a multi-slice two-dimensional (2D) MRI super-resolution network that combines a Generative Adversarial Network (GAN) with feature fusion and a pre-trained slice interpolation network to achieve three-dimensional (3D) super-resolution. The proposed model requires consecutively acquired three low-resolution (LR) MRI slices along a specific axis, and achieves the reconstruction of the MRI slices in the remaining two axes. The network effectively enhances both in-plane and out-of-plane resolution along the sagittal axis while addressing computational and memory constraints in 3D super-resolution. The proposed generator has a in-plane and out-of-plane Attention (IOA) network that fuses both in-plane and out-plane features of MRI dynamically. In terms of out-of-plane attention, the network merges features by considering the similarity distance between features and for in-plane attention, the network employs a two-level pyramid structure with varying receptive fields to extract features at different scales, ensuring the inclusion of both global and local features. Subsequently, to achieve 3D MRI super-resolution, a pre-trained slice interpolation network is used that takes two consecutive super-resolved MRI slices to generate a new intermediate slice. To further enhance the network performance and perceptual quality, we introduce a feature up-sampling layer and a feature extraction block with Scaled Exponential Linear Unit (SeLU). Moreover, our super-resolution network incorporates VGG loss from a fine-tuned VGG-19 network to provide additional enhancement. Through experimental evaluations on the IXI dataset and BRATS dataset, using the peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) and the number of training parameters, we demonstrate the superior performance of our method compared to the existing techniques. Also, the proposed model can be adapted or modified to achieve super-resolution for both 2D and 3D MRI data.

Study on Fine-Grained Visual Classification of Low-Resolution Urinary Erythrocyte

The morphological analysis test item of urine red blood cells is referred to as "extracorporeal renal biopsy," which holds significant importance for medical department testing. However, the accuracy of existing urine red blood cell morphology analyzers is suboptimal, and they are not widely utilized in medical examinations. Challenges include low image spatial resolution, blurred distinguishing features between cells, difficulty in fine-grained feature extraction, and insufficient data volume. This article aims to improve the classification accuracy of low-resolution urine red blood cells. This paper proposes a super-resolution method based on category-aware loss and an RBC-MIX data enhancement approach. It optimizes the cross-entropy loss to maximize the classification boundary and improve intra-class tightness and inter-class difference, achieving fine-grained classification of low-resolution urine red blood cells. Experimental outcomes demonstrate that with this method, an accuracy rate of 97.8% can be achieved for low-resolution urine red blood cell images. This algorithm attains outstanding classification performance for low-resolution urine red blood cells with only category labels required. This method can serve as a practical reference for urine red blood cell morphology examination items.

Numerical study of transient absorption saturation in single-layer graphene for optical nanoscopy applications

Transient absorption, or pump-probe microscopy is an absorption-based technique that can explore samples ultrafast dynamic properties and provide fluorescence-free contrast mechanisms. When applied to graphene and its derivatives, this technique exploits the graphene transient response caused by the ultrafast interband transition as the imaging contrast mechanism. The saturation of this transition is fundamental to allow for super-resolution optical far-field imaging, following the reversible saturable optical fluorescence transitions (RESOLFT) concept, although not involving fluorescence. With this aim, we propose a model to numerically compute the temporal evolution under saturation conditions of the single-layer graphene molecular states, which are involved in the transient absorption. Exploiting an algorithm based on the fourth order Runge-Kutta (RK4) method, and the density matrix approach, we numerically demonstrate that the transient absorption signal of single-layer graphene varies linearly as a function of excitation intensity until it reaches saturation. We experimentally verify this model using a custom pump-probe super-resolution microscope. The results define the intensities necessary to achieve super-resolution in a pump-probe nanoscope while studying graphene-based materials and open the possibility of predicting such a saturation process in other light-matter interactions that undergo the same transition.

Quasi-supervised learning for super-resolution PET

Low resolution of positron emission tomography (PET) limits its diagnostic performance. Deep learning has been successfully applied to achieve super-resolution PET. However, commonly used supervised learning methods in this context require many pairs of low- and high-resolution (LR and HR) PET images. Although unsupervised learning utilizes unpaired images, the results are not as good as that obtained with supervised deep learning. In this paper, we propose a quasi-supervised learning method, which is a new type of weakly-supervised learning methods, to recover HR PET images from LR counterparts by leveraging similarity between unpaired LR and HR image patches. Specifically, LR image patches are taken from a patient as inputs, while the most similar HR patches from other patients are found as labels. The similarity between the matched HR and LR patches serves as a prior for network construction. Our proposed method can be implemented by designing a new network or modifying an existing network. As an example in this study, we have modified the cycle-consistent generative adversarial network (CycleGAN) for super-resolution PET. Our numerical and experimental results qualitatively and quantitatively show the merits of our method relative to the state-of-the-art methods. The code is publicly available at https://github.com/PigYang-ops/CycleGAN-QSDL.

Deep local-to-global feature learning for medical image super-resolution

Medical images play a vital role in medical analysis by providing crucial information about patients' pathological conditions. However, the quality of these images can be compromised by many factors, such as limited resolution of the instruments, artifacts caused by movements, and the complexity of the scanned areas. As a result, low-resolution (LR) images cannot provide sufficient information for diagnosis. To address this issue, researchers have attempted to apply image super-resolution (SR) techniques to restore the high-resolution (HR) images from their LR counterparts. However, these techniques are designed for generic images, and thus suffer from many challenges unique to medical images. An obvious one is the diversity of the scanned objects; for example, the organs, tissues, and vessels typically appear in different sizes and shapes, and are thus hard to restore with standard convolution neural networks (CNNs). In this paper, we develop a dynamic-local learning framework to capture the details of these diverse areas, consisting of deformable convolutions with adjustable kernel shapes. Moreover, the global information between the tissues and organs is vital for medical diagnosis. To preserve global information, we propose pixel-pixel and patch-patch global learning using a non-local mechanism and a vision transformer (ViT), respectively. The result is a novel CNN-ViT neural network with Local-to-Global feature learning for medical image SR, referred to as LGSR, which can accurately restore both local details and global information. We evaluate our method on six public datasets and one large-scale private dataset, which include five different types of medical images (i.e., Ultrasound, OCT, Endoscope, CT, and MRI images). Experiments show that the proposed method achieves superior PSNR/SSIM and visual performance than the state of the arts with competitive computational costs, measured in network parameters, runtime, and FLOPs. What is more, the experiment conducted on OCT image segmentation for the downstream task demonstrates a significantly positive performance effect of LGSR.

Comparison of model-based versus deep learning-based image reconstruction for thin-slice T2-weighted spin-echo prostate MRI

PURPOSE

To compare a previous model-based image reconstruction (MBIR) with a newly developed deep learning (DL)-based image reconstruction for providing improved signal-to-noise ratio (SNR) in high through-plane resolution (1 mm) T2-weighted spin-echo (T2SE) prostate MRI.

METHODS

Large-area contrast and high-contrast spatial resolution of the reconstruction methods were assessed quantitatively in experimental phantom studies. The methods were next evaluated radiologically in 17 subjects at 3.0 Tesla for whom prostate MRI was clinically indicated. For each subject, the axial T2SE raw data were directed to MBIR and to the DL reconstruction at three vendor-provided levels: (L)ow, (M)edium, and (H)igh. Thin-slice images from the four reconstructions were compared using evaluation criteria related to SNR, sharpness, contrast fidelity, and reviewer preference. Results were compared using the Wilcoxon signed-rank test using Bonferroni correction, and inter-reader comparisons were done using the Cohen and Krippendorf tests.

RESULTS

Baseline contrast and resolution in phantom studies were equivalent for all four reconstruction pathways as desired. In vivo, all three DL levels (L, M, H) provided improved SNR versus MBIR. For virtually, all other evaluation criteria DL L and M were superior to MBIR. DL L and M were evaluated as superior to DL H in fidelity of contrast. For 44 of the 51 evaluations, the DL M reconstruction was preferred.

CONCLUSION

The deep learning reconstruction method provides significant SNR improvement in thin-slice (1 mm) T2SE images of the prostate while retaining image contrast. However, if taken to too high a level (DL High), both radiological sharpness and fidelity of contrast diminish.

Degeneration in Nucleus basalis of Meynert signals earliest stage of Alzheimer's disease progression

Nucleus Basalis of Meynert (NbM), a crucial source of cholinergic projection to the entorhinal cortex (EC) and hippocampus (HC), has shown sensitivity to neurofibrillary degeneration in the early stages of Alzheimer's Disease. Using deformation-based morphometry (DBM) on up-sampled MRI scans from 1447 Alzheimer's Disease Neuroimaging Initiative participants, we aimed to quantify NbM degeneration along the disease trajectory. Results from cross-sectional analysis revealed significant differences of NbM volume between cognitively normal and early mild cognitive impairment cohorts, confirming recent studies suggesting that NbM degeneration happens before degeneration in the EC or HC. Longitudinal linear mixed-effect models were then used to compare trajectories of volume change after realigning all participants into a common timeline based on their cognitive decline. Results indicated the earliest deviations in NbM volumes from the cognitively healthy trajectory, challenging the prevailing idea that Alzheimer's originates in the EC. Converging evidence from cross-sectional and longitudinal models suggest that the NbM may be a focal target of early AD progression, which is often obscured by normal age-related decline.

Nanoscale myelinogenesis image in developing brain via super-resolution nanoscopy by near-infrared emissive curcumin-BODIPY derivatives

Understanding the intricate nanoscale architecture of neuronal myelin during central nervous system development is of utmost importance. However, current visualization methods heavily rely on electron microscopy or indirect fluorescent method, lacking direct and real-time imaging capabilities. Here, we introduce a breakthrough near-infrared emissive curcumin-BODIPY derivative (MyL-1) that enables direct visualization of myelin structure in brain tissues. The remarkable compatibility of MyL-1 with stimulated emission depletion nanoscopy allows for unprecedented super-resolution imaging of myelin ultrastructure. Through this innovative approach, we comprehensively characterize the nanoscale myelinogenesis in three dimensions over the course of brain development, spanning from infancy to adulthood in mouse models. Moreover, we investigate the correlation between myelin substances and Myelin Basic Protein (MBP), shedding light on the essential role of MBP in facilitating myelinogenesis during vertebral development. This novel material, MyL-1, opens up new avenues for studying and understanding the intricate process of myelinogenesis in a direct and non-invasive manner, paving the way for further advancements in the field of nanoscale neuroimaging.

Learn from orientation prior for radiograph super-resolution: Orientation operator transformer

BACKGROUND AND OBJECTIVE

High-resolution radiographic images play a pivotal role in the early diagnosis and treatment of skeletal muscle-related diseases. It is promising to enhance image quality by introducing single-image super-resolution (SISR) model into the radiology image field. However, the conventional image pipeline, which can learn a mixed mapping between SR and denoising from the color space and inter-pixel patterns, poses a particular challenge for radiographic images with limited pattern features. To address this issue, this paper introduces a novel approach: Orientation Operator Transformer - O2former.

METHODS

We incorporate an orientation operator in the encoder to enhance sensitivity to denoising mapping and to integrate orientation prior. Furthermore, we propose a multi-scale feature fusion strategy to amalgamate features captured by different receptive fields with the directional prior, thereby providing a more effective latent representation for the decoder. Based on these innovative components, we propose a transformer-based SISR model, i.e., O2former, specifically designed for radiographic images.

RESULTS

The experimental results demonstrate that our method achieves the best or second-best performance in the objective metrics compared with the competitors at ×4 upsampling factor. For qualitative, more objective details are observed to be recovered.

CONCLUSIONS

In this study, we propose a novel framework called O2former for radiological image super-resolution tasks, which improves the reconstruction model's performance by introducing an orientation operator and multi-scale feature fusion strategy. Our approach is promising to further promote the radiographic image enhancement field.

Colocalization by cross-correlation, a new method of colocalization suited for super-resolution microscopy

BACKGROUND

A common goal of scientific microscopic imaging is to determine if a spatial correlation exists between two imaged structures. This is generally accomplished by imaging fluorescently labeled structures and measuring their spatial correlation with a class of image analysis algorithms known as colocalization. However, the most commonly used methods of colocalization have strict limitations, such as requiring overlap in the fluorescent markers and reporting requirements for accurate interpretation of the data, that are often not met. Due to the development of novel super-resolution techniques, which reduce the overlap of the fluorescent signals, a new colocalization method is needed that does not have such strict requirements.

RESULTS

In order to overcome the limitations of other colocalization algorithms, I developed a new ImageJ/Fiji plugin, Colocalization by cross-correlation (CCC). This method uses cross-correlation over space to identify spatial correlations as a function of distance, removing the overlap requirement and providing more comprehensive results. CCC is compatible with 3D and time-lapse images, and was designed to be easy to use. CCC also generates new images that only show the correlating labeled structures from the input images, a novel feature among the cross-correlating algorithms.

CONCLUSIONS

CCC is a versatile, powerful, and easy to use colocalization and spatial correlation tool that is available through the Fiji update sites. Full and up to date documentation can be found at https://imagej.net/plugins/colocalization-by-cross-correlation . CCC source code is available at https://github.com/andmccall/Colocalization_by_Cross_Correlation .

Model-based super-resolution reconstruction for pseudo-continuous Arterial Spin Labeling

Arterial spin labeling (ASL) is a promising, non-invasive perfusion magnetic resonance imaging technique for quantifying cerebral blood flow (CBF). Unfortunately, ASL suffers from an inherently low signal-to-noise ratio (SNR) and spatial resolution, undermining its potential. Increasing spatial resolution without significantly sacrificing SNR or scan time represents a critical challenge towards routine clinical use. In this work, we propose a model-based super-resolution reconstruction (SRR) method with joint motion estimation that breaks the traditional SNR/resolution/scan-time trade-off. From a set of differently oriented 2D multi-slice pseudo-continuous ASL images with a low through-plane resolution, 3D-isotropic, high resolution, quantitative CBF maps are estimated using a Bayesian approach. Experiments on both synthetic whole brain phantom data, and on in vivo brain data, show that the proposed SRR Bayesian estimation framework outperforms state-of-the-art ASL quantification.

CVANet: Cascaded visual attention network for single image super-resolution

Deep convolutional neural networks (DCNNs) have exhibited excellent feature extraction and detail reconstruction capabilities for single image super-resolution (SISR). Nevertheless, most previous DCNN-based methods do not fully utilize the complementary strengths between feature maps, channels, and pixels. Therefore, it hinders the ability of DCNNs to represent abundant features. To tackle the aforementioned issues, we present a Cascaded Visual Attention Network for SISR called CVANet, which simulates the visual attention mechanism of the human eyes to focus on the reconstruction process of details. Specifically, we first designed a trainable feature attention module (FAM) for feature-level attention learning. Afterward, we introduce a channel attention module (CAM) to reinforce feature maps under channel-level attention learning. Meanwhile, we propose a pixel attention module (PAM) that adaptively selects representative features from the previous layers, which are utilized to generate a high-resolution image. Satisfactory, our CVANet can effectively improve the resolution of images by exploring the feature representation capabilities of different modules and the visual perception properties of the human eyes. Extensive experiments with different methods on four benchmarks demonstrate that our CVANet outperforms the state-of-the-art (SOTA) methods in subjective visual perception, PSNR, and SSIM.The code will be made available https://github.com/WilyZhao8/CVANet.

Stereo attention-based all-in-one super-resolution for robot-assisted minimally invasive surgery

Robot-assisted minimally invasive surgery (MIS) faces challenges in obtaining high-quality imaging results due to the limited spatial environment. In this paper, we present an all-in-one image super-resolution (SR) algorithm designed to tackle this challenge. By utilizing the stereo information from binocular images, we effectively convert low-resolution images into high-resolution ones. Our model architecture amalgamates the prowess of Convolutional Neural Networks (CNNs) and Transformers, capitalizing on the advantages of both methodologies. To achieve super-resolution across all scale factors, we employ a trainable upsampling module within our proposed network. We substantiate the effectiveness of our method through extensive quantitative and qualitative experiments. The results of our evaluations provide strong evidence supporting the superior performance of our approach in enhancing the quality of surgical images. Our method improves the resolution and thus the overall image quality, which allows the surgeon to perform precise operations conveniently. Simultaneously, it also facilitates the scaling of the region of interest (ROI) to achieve high-quality visualization during surgical procedures. Furthermore, it has the potential to enhance the image quality during telesurgery.

An efficient multi-scale learning method for image super-resolution networks

The image super-resolution (SR) operation holds multiple solutions with the one-to-many mapping from low-resolution (LR) to high-resolution (HR) space. However, the SR of different scales for the same image is usually regarded as independent tasks in the existing SR networks. Therefore, these networks are inflexible to effectively utilize feature learning experience and require much more computing time to recover HR images in higher resolutions. Recent arbitrary scale SR methods still cannot solve these problems. To efficiently and effectively recover HR images, this paper presents an efficient multi-scale learning method for image SR networks based on a novel self-generating (SG) mechanism. This method (briefly named SG-SR) utilizes the feature learning results of SR networks to generate upscale filters by using the novel SG upscale module, which is proposed to replace the traditional upscale module. For each scale factor, the SG upscale module provides the corresponding amount of the spatial weights to filter the LR tensor and then converts filtered tensors with the original tensor to corresponding HR images. The proposed method is evaluated through extensive experiments and compared with state-of-the-art (SOTA) methods on widely used benchmark datasets. The experimental results show that our method has superior performance compared with SOTA methods, and the SG upscale module can improve the performance of existing SR networks effectively. What is more, our module has a much less calculation cost than the other upscale modules.

Multi-scale feature selection network for lightweight image super-resolution

Recently, many super-resolution (SR) methods based on convolutional neural networks (CNNs) have achieved superior performance by utilizing deep and heavy models, which may not be suitable for real-world low-budget devices. To address this issue, we propose a novel lightweight SR network called a multi-scale feature selection network (MFSN). As the basic building block of MFSN, the multi-scale feature selection block (MFSB) is presented to extract the rich multi-scale features from a coarse-to-fine receptive field level. For a better representation ability, a wide-activated residual unit is adopted in each branch of MFSB except the last one. In MFSB, the scale selection module (SSM) is designed to effectively fuse the features from two adjacent branches by adjusting receptive field sizes adaptively. Further, a comprehensive channel attention mechanism (CCAM) is integrated into SSM to learn the dynamic selection weight by considering the local and global inter-channel dependencies. Extensive experimental results illustrate that the proposed MFSN is superior to other lightweight methods.

Ultrasound Localization Microscopy for Breast Cancer Imaging in Patients: Protocol Optimization and Comparison with Shear Wave Elastography

OBJECTIVE

Ultrasound localization microscopy (ULM) has gained increasing attention in recent years because of its ability to visualize blood vessels at super-resolution. The field of oncology, in particular, could benefit from detailed vascular characterization, for example, for diagnosis and therapy monitoring. This study was aimed at refining ULM for breast cancer patients by optimizing the measurement protocol, identifying translational challenges and combining ULM and shear wave elastography.

METHODS

We computed ULM images of 11 patients with breast cancer by recording contrast-enhanced ultrasound (CEUS) sequences and post-processing them in an offline pipeline. For CEUS, two different doses and injection speeds of SonoVue were applied. The best injection protocol was determined based on quantitative parameters derived from so-called occurrence maps. In addition, a suitable measurement time window was determined, also considering the occurrence of motion. ULM results were compared with shear wave elastography and histological vessel density.

RESULTS

At the higher dose and injection speed, the highest number of microbubbles, number of tracks and vessel coverage were achieved, leading to the most detailed representation of tumor vasculature. Even at the highest concentration, no significant overlay of microbubble signals occurred. Motion significantly reduced the number of usable frames, thus limiting the measurement window to 3.5 min. ULM vessel coverage was comparable to the histological vessel fraction and correlated significantly with mean tumor elasticity.

CONCLUSION

The settings for microbubble injection strongly influence ULM images, thus requiring optimized protocols for different indications. Patient and examiner motion was identified as the main translational challenge for ULM.

Advanced optical imaging for the rational design of nanomedicines

Despite the enormous potential of nanomedicines to shape the future of medicine, their clinical translation remains suboptimal. Translational challenges are present in every step of the development pipeline, from a lack of understanding of patient heterogeneity to insufficient insights on nanoparticle properties and their impact on material-cell interactions. Here, we discuss how the adoption of advanced optical microscopy techniques, such as super-resolution optical microscopies, correlative techniques, and high-content modalities, could aid the rational design of nanocarriers, by characterizing the cell, the nanomaterial, and their interaction with unprecedented spatial and/or temporal detail. In this nanomedicine arena, we will discuss how the implementation of these techniques, with their versatility and specificity, can yield high volumes of multi-parametric data; and how machine learning can aid the rapid advances in microscopy: from image acquisition to data interpretation.

Lattice Light-Sheet Motor-PAINT: A Method to Map the Orientations of Microtubules in Complex Three-Dimensional Arrays

Microtubules play an essential role in many cellular functions, in part by serving as tracks for intracellular transport by kinesin and dynein. The ability of microtubules to fulfill this role fundamentally depends on the fact that they are polar, with motors moving along them toward either their plus or minus end. Given that the microtubule cytoskeleton adopts a variety of specialized architectures in different cell types, it is important to map precisely how microtubules are oriented and organized in these cells. To this end, motor-PAINT has been developed, but in its current implementation, it relies on total internal reflection fluorescence (TIRF) microscopy and is thus restricted to imaging microtubules in a thin section of the cell immediately adjacent to the coverslip. Here, we report a variant of motor-PAINT that uses lattice light-sheet microscopy to overcome this, allowing for the mapping of microtubule organization and orientation in three-dimensional samples. We describe the necessary steps to purify, label, use, and image kinesin motors for motor-PAINT and outline the analysis pipeline used to visualize the resulting data. The method described here can be used in the future to study the microtubule cytoskeleton in (thick) polarized cells such as intestinal epithelial cells.

Super-resolution fluorescence imaging of cryosamples does not limit achievable resolution in cryoEM

Correlated super-resolution cryo-fluorescence and cryo-electron microscopy (cryoEM) has been gaining popularity as a method to investigate biological samples with high resolution and specificity. A concern in this combined method (called SR-cryoCLEM), however, is whether and how fluorescence imaging prior to cryoEM acquisition is detrimental to sample integrity. In this report, we investigated the effect of high-dose laser light (405, 488, and 561 nm) irradiation on apoferritin samples prepared for cryoEM with excitation wavelengths commonly used in fluorescence microscopy, and compared these samples to controls that were kept in the dark. We found that laser illumination, of equal duration and intensity as used in cryo-single molecule localization microscopy (cryoSMLM) and in the presence of high concentrations of fluorescent protein, did not affect the achievable resolution in cryoEM, with final reconstructions reaching resolutions of ∼ 1.8 Å regardless of the laser illumination. The finding that super-resolution fluorescence imaging of cryosamples prior to cryoEM data acquisition does not limit the achievable resolution suggests that super-resolution cryo-fluorescence microscopy and in situ structural biology using cryoEM are entirely compatible.

MGDUN: An interpretable network for multi-contrast MRI image super-resolution reconstruction

Magnetic resonance imaging (MRI) Super-Resolution (SR) aims to obtain high resolution (HR) images with more detailed information for precise diagnosis and quantitative image analysis. Deep unfolding networks outperform general MRI SR reconstruction methods by providing better performance and improved interpretability, which enhance the trustworthiness required in clinical practice. Additionally, current SR reconstruction techniques often rely on a single contrast or a simple multi-contrast fusion mechanism, ignoring the complex relationships between different contrasts. To address these issues, in this paper, we propose a Model-Guided multi-contrast interpretable Deep Unfolding Network (MGDUN) for medical image SR reconstruction, which explicitly incorporates the well-studied multi-contrast MRI observation model into an unfolding iterative network. Specifically, we manually design an objective function for MGDUN that can be iteratively computed by the half-quadratic splitting algorithm. The iterative MGDUN algorithm is unfolded into a special model-guided deep unfolding network that explicitly takes into account both the multi-contrast relationship matrix and the MRI observation matrix during the end-to-end optimization process. Extensive experimental results on the multi-contrast IXI dataset and the BraTs 2019 dataset demonstrate the superiority of our proposed model, with PSNR reaching 37.3366 and 35.9690 respectively. Our proposed MGDUN provides a promising solution for multi-contrast MR image super-resolution reconstruction. Code is available at https://github.com/yggame/MGDUN.

Super-resolution QSM in little or no additional time for imaging (NATIve) using 2D EPI imaging in 3 orthogonal planes

Quantitative Susceptibility Mapping has the potential to provide additional insights into neurological diseases but is typically based on a quite long (5-10 min) 3D gradient-echo scan which is highly sensitive to motion. We propose an ultra-fast acquisition based on three orthogonal (sagittal, coronal and axial) 2D simultaneous multi-slice EPI scans with 1 mm in-plane resolution and 3 mm thick slices. Images in each orientation are corrected for susceptibility-related distortions and co-registered with an iterative non-linear Minimum Deformation Averaging (Volgenmodel) approach to generate a high SNR, super-resolution data set with an isotropic resolution of close to 1 mm. The net acquisition time is 3 times the volume acquisition time of EPI or about 12 s, but the three volumes could also replace "dummy scans" in fMRI, making it feasible to acquire QSM in little or No Additional Time for Imaging (NATIve). NATIve QSM values agreed well with reference 3D GRE QSM in the basal ganglia in healthy subjects. In patients with multiple sclerosis, there was also a good agreement between the susceptibility values within lesions and control ROIs and all lesions which could be seen on 3D GRE QSMs could also be visualized on NATIve QSMs. The approach is faster than conventional 3D GRE by a factor of 25-50 and faster than 3D EPI by a factor of 3-5. As a 2D technique, NATIve QSM was shown to be much more robust to motion than the 3D GRE and 3D EPI, opening up the possibility of studying neurological diseases involving iron accumulation and demyelination in patients who find it difficult to lie still for long enough to acquire QSM data with conventional methods.

Super-resolution of 2D ultrasound images and videos

This paper proposes a novel deep-learning framework for super-resolution ultrasound images and videos in terms of spatial resolution and line reconstruction. To this end, we up-sample the acquired low-resolution image through a vision-based interpolation method; then, we train a learning-based model to improve the quality of the up-sampling. We qualitatively and quantitatively test our model on different anatomical districts (e.g., cardiac, obstetric) images and with different up-sampling resolutions (i.e., 2X, 4X). Our method improves the PSNR median value with respect to SOTA methods of [Formula: see text] on obstetric 2X raw images, [Formula: see text] on cardiac 2X raw images, and [Formula: see text] on abdominal raw 4X images; it also improves the number of pixels with a low prediction error of [Formula: see text] on obstetric 4X raw images, [Formula: see text] on cardiac 4X raw images, and [Formula: see text] on abdominal 4X raw images. The proposed method is then applied to the spatial super-resolution of 2D videos, by optimising the sampling of lines acquired by the probe in terms of the acquisition frequency. Our method specialises trained networks to predict the high-resolution target through the design of the network architecture and the loss function, taking into account the anatomical district and the up-sampling factor and exploiting a large ultrasound data set. The use of deep learning on large data sets overcomes the limitations of vision-based algorithms that are general and do not encode the characteristics of the data. Furthermore, the data set can be enriched with images selected by medical experts to further specialise the individual networks. Through learning and high-performance computing, the proposed super-resolution is specialised to different anatomical districts by training multiple networks. Furthermore, the computational demand is shifted to centralised hardware resources with a real-time execution of the network's prediction on local devices.

A super-resolution network for medical imaging via transformation analysis of wavelet multi-resolution

In recent years, deep learning super-resolution models for progressive reconstruction have achieved great success. However, these models which refer to multi-resolution analysis basically ignore the information contained in the lower subspaces and do not explore the correlation between features in the wavelet and spatial domain, resulting in not fully utilizing the auxiliary information brought by multi-resolution analysis with multiple domains. Therefore, we propose a super-resolution network based on the wavelet multi-resolution framework (WMRSR) to capture the auxiliary information contained in multiple subspaces and to be aware of the interdependencies between spatial domain and wavelet domain features. Initially, the wavelet multi-resolution input (WMRI) is generated by combining wavelet sub-bands obtained from each subspace through wavelet multi-resolution analysis and the corresponding spatial domain image content, which serves as input to the network. Then, the WMRSR captures the corresponding features from the WMRI in the wavelet domain and spatial domain, respectively, and fuses them adaptively, thus learning fully explored features in multi-resolution and multi-domain. Finally, the high-resolution images are gradually reconstructed in the wavelet multi-resolution framework by our convolution-based wavelet transform module which is suitable for deep neural networks. Extensive experiments conducted on two public datasets demonstrate that our method outperforms other state-of-the-art methods in terms of objective and visual qualities.

Reconstruction and completion of high-resolution 3D cardiac shapes using anisotropic CMRI segmentations and continuous implicit neural representations

Since the onset of computer-aided diagnosis in medical imaging, voxel-based segmentation has emerged as the primary methodology for automatic analysis of left ventricle (LV) function and morphology in cardiac magnetic resonance images (CMRI). In standard clinical practice, simultaneous multi-slice 2D cine short-axis MR imaging is performed under multiple breath-holds resulting in highly anisotropic 3D images. Furthermore, sparse-view CMRI often lacks whole heart coverage caused by large slice thickness and often suffers from inter-slice misalignment induced by respiratory motion. Therefore, these volumes only provide limited information about the true 3D cardiac anatomy which may hamper highly accurate assessment of functional and anatomical abnormalities. To address this, we propose a method that learns a continuous implicit function representing 3D LV shapes by training an auto-decoder. For training, high-resolution segmentations from cardiac CT angiography are used. The ability of our approach to reconstruct and complete high-resolution shapes from manually or automatically obtained sparse-view cardiac shape information is evaluated by using paired high- and low-resolution CMRI LV segmentations. The results show that the reconstructed LV shapes have an unconstrained subvoxel resolution and appear smooth and plausible in through-plane direction. Furthermore, Bland-Altman analysis reveals that reconstructed high-resolution ventricle volumes are closer to the corresponding reference volumes than reference low-resolution volumes with bias of [limits of agreement] -3.51 [-18.87, 11.85] mL, and 12.96 [-10.01, 35.92] mL respectively. Finally, the results demonstrate that the proposed approach allows recovering missing shape information and can indirectly correct for limited motion-induced artifacts.

MHAN: Multi-Stage Hybrid Attention Network for MRI reconstruction and super-resolution

High-quality magnetic resonance imaging (MRI) affords clear body tissue structure for reliable diagnosing. However, there is a principal problem of the trade-off between acquisition speed and image quality. Image reconstruction and super-resolution are crucial techniques to solve these problems. In the main field of MR image restoration, most researchers mainly focus on only one of these aspects, namely reconstruction or super-resolution. In this paper, we propose an efficient model called Multi-Stage Hybrid Attention Network (MHAN) that performs the multi-task of recovering high-resolution (HR) MR images from low-resolution (LR) under-sampled measurements. Our model is highlighted by three major modules: (i) an Amplified Spatial Attention Block (ASAB) capable of enhancing the differences in spatial information, (ii) a Self-Attention Block with a Data-Consistency Layer (DC-SAB), which can improve the accuracy of the extracted feature information, (iii) an Adaptive Local Residual Attention Block (ALRAB) that focuses on both spatial and channel information. MHAN employs an encoder-decoder architecture to deeply extract contextual information and a pipeline to provide spatial accuracy. Compared with the recent multi-task model T2Net, our MHAN improves by 2.759 dB in PSNR and 0.026 in SSIM with scaling factor ×2 and acceleration factor 4× on T2 modality.

Super-resolution of magnetic resonance images using Generative Adversarial Networks

Magnetic Resonance Imaging (MRI) typically comes at the cost of small spatial coverage, high expenses and long scan times. Accelerating MRI acquisition by taking less measurements yields the potential to relax these inherent forfeits. Recent breakthroughs in the field of Machine Learning have shown high-resolution (HR) images could be recovered from low-resolution (LR) signals via super-resolution (SR). In particular, a novel class of neural networks named Generative Adversarial Networks (GAN) has manifested an alternative way of conceiving models capable of generating data. GANs can learn to infer details based on some prior information, subsequently recovering missing data. Accordingly, they manifest huge potential in MRI reconstruction and acceleration tasks. This paper conducts a review on GAN-based SR methods, exhibiting the immersive ability of GANs on upscaling MRIs by a scale factor of ×4 while at the same time maintaining trustworthy and high-frequency details. Despite quantitative results suggesting SRResCycGAN outperforms other popular deep learning methods in recovering ×4 downgraded images, qualitative results show Beby-GAN holds the best perceptual quality and proves GAN-based methods hold the capacity to reduce medical costs, distress patients and even enable new MRI applications where it is currently too slow or expensive.

Ultrasound localization microscopy

Ultrasound Localization Microscopy (ULM) is an emerging technique that provides impressive super-resolved images of microvasculature, i.e., images with much better resolution than the conventional diffraction-limited ultrasound techniques and is already taking its first steps from preclinical to clinical applications. In comparison to the established perfusion or flow measurement methods, namely contrast-enhanced ultrasound (CEUS) and Doppler techniques, ULM allows imaging and flow measurements even down to the capillary level. As ULM can be realized as a post-processing method, conventional ultrasound systems can be used for. ULM relies on the localization of single microbubbles (MB) of commercial, clinically approved contrast agents. In general, these very small and strong scatterers with typical radii of 1-3 µm are imaged much larger in ultrasound images than they actually are due to the point spread function of the imaging system. However, by applying appropriate methods, these MBs can be localized with sub-pixel precision. Then, by tracking MBs over successive frames of image sequences, not only the morphology of vascular trees but also functional information such as flow velocities or directions can be obtained and visualized. In addition, quantitative parameters can be derived to describe pathological and physiological changes in the microvasculature. In this review, the general concept of ULM and conditions for its applicability to microvessel imaging are explained. Based on this, various aspects of the different processing steps for a concrete implementation are discussed. The trade-off between complete reconstruction of the microvasculature and the necessary measurement time as well as the implementation in 3D are reviewed in more detail, as they are the focus of current research. Through an overview of potential or already realized preclinical and clinical applications - pathologic angiogenesis or degeneration of vessels, physiological angiogenesis, or the general understanding of organ or tissue function - the great potential of ULM is demonstrated.

Super-resolution ultrasound microvascular imaging: Is it ready for clinical use?

The field of super-resolution ultrasound microvascular imaging has been rapidly growing over the past decade. By leveraging contrast microbubbles as point targets for localization and tracking, super-resolution ultrasound pinpoints the location of microvessels and measures their blood flow velocity. Super-resolution ultrasound is the first in vivo imaging modality that can image micron-scale vessels at a clinically relevant imaging depth without tissue destruction. These unique capabilities of super-resolution ultrasound provide structural (vessel morphology) and functional (vessel blood flow) assessments of tissue microvasculature on a global and local scale, which opens new doors for many enticing preclinical and clinical applications that benefit from microvascular biomarkers. The goal of this short review is to provide an update on recent advancements in super-resolution ultrasound imaging, with a focus on summarizing existing applications and discussing the prospects of translating super-resolution imaging to clinical practice and research. In this review, we also provide brief introductions of how super-resolution ultrasound works, how does it compare with other imaging modalities, and what are the tradeoffs and limitations for an audience who is not familiar with the technology.

Unique features of the arterial blood-brain barrier

CNS vasculature differs from vascular networks of peripheral organs by its ability to tightly control selective material exchange across capillary barriers. Capillary permeability is mostly defined by unique cellular components of the endothelium. While capillaries are extensively investigated, the barrier properties of larger vessels are understudied. Here, we investigate barrier properties of CNS arterial walls. Using tracer challenges and various imaging modalities, we discovered that at the mouse cortex, the arterial barrier does not reside at the classical level of the endothelium. The arterial wall's unique permeability acts bi-directionally; CSF substances travel along the glymphatic path and can penetrate from the peri-vascular space through arteriolar walls towards the lumen. We found that caveolae vesicles in arteriole endothelial are functional transcytosis machinery components, and that a similar mechanism is evident in the human brain. Our discoveries highlight vascular heterogeneity investigations as a potent approach to uncover new barrier mechanisms.

Optimization technique for increasing resolution in computed tomography imaging

Starting from the importance of conforming to biological reality in medicine, in this paper we propose an optimization technique for increasing resolution of computed tomography (CT) images acquired using various existing scanners. Considering a three-dimensional Hounsfield Units (HU) array, together with the corresponding spatial metadata of interest (pixel sizes and slice thickness), the procedure is based on halving each voxel along the directions of the device's Cartesian frame of reference and find those values which are both satisfying the X-Rays attenuation coefficient average requirement and minimizing the HU distance to classical interpolation points. The discussed method was tested by implementing a C# .Net 6, cross-platform library containing two algorithm flavors that could be independently applied: "Z" for doubling the number of slices, and "XY" for doubling the resolution of individual slices. This design allows also chaining (e.g. one could apply the "Z,XY,Z" sequence in order to reduce four times slice thickness). In the context of existing unavoidable limitations, the first results are suggesting the "CT compatible" interpolation technique could provide a reasonable approximation of reality. However, the main advantage comes from satisfying mass conservation, which is of high importance in medical diagnosis and treatment.•The Hounsfield Units scale is defined as a linear transformation of the X-Rays attenuation coefficients. Thus, splitting a computed tomography voxel into two congruent volumes must satisfy the HU average requirement (the initial value must equal the average of the two output HU values).•Existing interpolation methods (linear, spline, etc.) are not compatible with the computed tomography HU average requirement. This could also result in mass estimate anomalies with significant impact in medical diagnosis.•The proposed "CT compatible" interpolation method is based on finding those values which are both satisfying the X-Rays attenuation coefficient average requirement and minimizing the Hounsfield Units distance to classical interpolation points.

MRIFlow: Magnetic resonance image super-resolution based on normalizing flow and frequency prior

Super-resolution (SR) is a computer vision task that involves recovering high-resolution (HR) images from low-resolution (LR) ones. While SR is applied to various disciplines, it is particularly important in the medical field which requires accurate diagnosis. L1 and L2 loss-based SR methods produce high values for the peak signal-to-noise ratio and structural similarity index measure but do not have high perceptual quality because SR methods are trained with the average of plausible HR predictions. In addition, SR is an ill-posed problem because only one LR image can be mapped to various HR images. This is crucial because poorly generated HR images can lead to misdiagnosis. In this paper, we propose MRIFlow, a novel method based on normalizing flow that transforms LR magnetic resonance (MR) images into HR MR images. MRIFlow contains frequency affine injectors to reflect frequency information. The frequency affine injector receives the output of a pre-trained LR encoder as the input and obtains frequency information from a wavelet transform based on ScatterNet. Using this method, its inverse operation is possible. MRIFlow has two versions based on the type of ScatterNet employed. In this paper, MRIFlow is compared with normalizing flow-based SR methods by using various MR image datasets such as IXI dataset, NYU fastMRI dataset, and LGG dataset and is demonstrated to produce better quantitative and qualitative results.

Multivariable-incorporating super-resolution residual network for transcranial focused ultrasound simulation

BACKGROUND AND OBJECTIVE

Transcranial focused ultrasound (tFUS) has emerged as a new non-invasive brain stimulation (NIBS) modality, with its exquisite ability to reach deep brain areas at a high spatial resolution. Accurate placement of an acoustic focus to a target region of the brain is crucial during tFUS treatment; however, the distortion of acoustic wave propagation through the intact skull casts challenges. High-resolution numerical simulation allows for monitoring of the acoustic pressure field in the cranium but also demands extensive computational loads. In this study, we adopt a super-resolution residual network technique based on a deep convolution to enhance the prediction quality of the FUS acoustic pressure field in the targeted brain regions.

METHODS

The training dataset was acquired by numerical simulations performed at low-(1.0 mm) and high-resolutions (0.5mm) on three ex vivo human calvariae. Five different super-resolution (SR) network models were trained by using a multivariable dataset in 3D, which incorporated information on the acoustic pressure field, wave velocity, and localized skull computed tomography (CT) images.

RESULTS

The accuracy of 80.87±4.50% in predicting the focal volume with a substantial improvement of 86.91% in computational cost compared to the conventional high-resolution numerical simulation was achieved. The results suggest that the method can greatly reduce the simulation time without sacrificing accuracy and improve the accuracy further with the use of additional inputs.

CONCLUSIONS

In this research, we developed multivariable-incorporating SR neural networks for transcranial focused ultrasound simulation. Our super-resolution technique may contribute to promoting the safety and efficacy of tFUS-mediated NIBS by providing on-site feedback information on the intracranial pressure field to the operator.

Deep learning enables super-resolution hydrodynamic flooding process modeling under spatiotemporally varying rainstorms

Real-time information on flooding extent, severity, and duration is necessary for effective metropolitan flood emergency management. Existing pluvial flood analysis methods are unable to simulate real-time regional flooding processes under spatiotemporally varying rainstorms. This paper presents a deep learning-enabled super-resolution hydrodynamic flood analysis method to simulate the real-time pluvial flooding process over a large area under spatiotemporally varying rainstorms. Compared with existing flood downscaling techniques, which are limited to flow depth, the proposed method produces high-resolution flow depth and velocity predictions, providing more comprehensive information for flood emergency management. The proposed method adopts a coarse-grid hydrodynamic model to generate a low-resolution flood map time series, which is subsequently converted to high-resolution flood maps by a deep learning model. The deep learning model can be trained using a limited number of assumed rainfall scenarios, which greatly reduces data preparation effort. The proposed method is applied to a complex terrain of 352 km² in Hong Kong that covers both mountainous and urban areas. Results show that the proposed method simulates the spatiotemporal variations of flood depth and velocity with root mean square errors as low as 0.082 m and 0.088 m/s, respectively, and correlation coefficients of 0.962 and 0.921, respectively. The computation time for a 48-h rainfall event in the study area is less than 30 s, which is 2690 times faster than the direct fine-grid hydrodynamic analysis. The deep learning-enabled super-resolution hydrodynamic flood analysis method provides a promising computational tool for emergency flood risk management.

Medical image super-resolution reconstruction algorithms based on deep learning: A survey

BACKGROUND AND OBJECTIVE

With the high-resolution (HR) requirements of medical images in clinical practice, super-resolution (SR) reconstruction algorithms based on low-resolution (LR) medical images have become a research hotspot. This type of method can significantly improve image SR without improving hardware equipment, so it is of great significance to review it.

METHODS

Aiming at the unique SR reconstruction algorithms in the field of medical images, based on subdivided medical fields such as magnetic resonance (MR) images, computed tomography (CT) images, and ultrasound images. Firstly, we deeply analyzed the research progress of SR reconstruction algorithms, and summarized and compared the different types of algorithms. Secondly, we introduced the evaluation indicators corresponding to the SR reconstruction algorithms. Finally, we prospected the development trend of SR reconstruction technology in the medical field.

RESULTS

The medical image SR reconstruction technology based on deep learning can provide more abundant lesion information, relieve the expert's diagnosis pressure, and improve the diagnosis efficiency and accuracy.

CONCLUSION

The medical image SR reconstruction technology based on deep learning helps to improve the quality of medicine, provides help for the diagnosis of experts, and lays a solid foundation for the subsequent analysis and identification tasks of the computer, which is of great significance for improving the diagnosis efficiency of experts and realizing intelligent medical care.

ESPRESO: An algorithm to estimate the slice profile of a single magnetic resonance image

To reduce scan time, magnetic resonance (MR) images are often acquired using 2D multi-slice protocols with thick slices that may also have gaps between them. The resulting image volumes have lower resolution in the through-plane direction than in the in-plane direction, and the through-plane resolution is in part characterized by the protocol's slice profile which acts as a through-plane point spread function (PSF). Although super-resolution (SR) has been shown to improve the visualization and down-stream processing of 2D multi-slice MR acquisitions, previous algorithms are usually unaware of the true slice profile, which may lead to sub-optimal SR performance. In this work, we present an algorithm to estimate the slice profile of a 2D multi-slice acquisition given only its own image volume without any external training data. We assume that an anatomical image is isotropic in the sense that, after accounting for a correctly estimated slice profile, the image patches along different orientations have the same probability distribution. Our proposed algorithm uses a modified generative adversarial network (GAN) where the generator network estimates the slice profile to reduce the resolution of the in-plane direction, and the discriminator network determines whether a direction is generated or real low resolution. The proposed algorithm, ESPRESO, which stands for "estimating the slice profile for resolution enhancement of a single image only", was tested with a state-of-the-art internally supervised SR algorithm. Specifically, ESPRESO is used to create training data for this SR algorithm, and results show improvements when ESPRESO is used over commonly-used PSFs.

Feedback attention network for cardiac magnetic resonance imaging super-resolution

BACKGROUND AND OBJECTIVE

Atrial fibrillation (AF) is a common clinical arrhythmia with a high disability and mortality rate. Improving the resolution of atrial structure and its changes in patients with AF is very important for understanding and treating AF.

METHODS

Aiming at the problems of previous deep learning-based image super-resolution (SR) reconstruction methods simply deepening the network, loss of upsampling information, and difficulty in the reconstruction of high-frequency information, we propose the Feedback Attention Network (FBAN) for cardiac magnetic resonance imaging (CMRI) super-resolution. The network comprises a preprocessing module, a multi-scale residual group module, an upsampling module, and a reconstruction module. The preprocessing module uses a convolutional layer to extract shallow features and dilate the number of channels of the feature map. The multi-scale residual group module adds a multi-channel network, a mixed attention mechanism, and a long and short skip connection to expand the receptive field of the feature map, improve the multiplexing of multi-scale features and strengthen the reconstruction of high-frequency information. The upsampling module adopts the sub-pixel method to upsample the feature map to the target image size. The reconstruction module consists of a convolutional layer, which is used to restore the number of channels of the feature map to the original number to obtain the reconstructed high-resolution (HR) image.

RESULTS

Furthermore, the test results on the public dataset of CMRI show that the HR images reconstructed by the FBAN method not only have a good improvement in reconstructed edge and texture information but also have a good improvement in the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) objective evaluation indicators.

CONCLUSION

Compared with the local magnified image, the edge information of the FBAN method reconstructed image has been enhanced, more high-frequency information of the CMRI is restored, the texture details are less lost, and the reconstructed image is less blurry. Overall, the reconstructed image has a lighter feeling of smearing, and the visual experience is more apparent and sharper.

Quaternion attention multi-scale widening network for endoscopy image super-resolution

OBJECTIVE

In the field of endoscopic imaging, Super-Resolution (SR) plays an important role in Manufactured Diagnosis, physicians and machine Automatic Diagnosis. Although many recent studies have been performed, by using deep convolutional neural networks on endoscopic Super-Resolution, most of the methods have large parameters, which limits their practical application. In addition, almost all of these methods treat each channel equally based on the real-valued domain, without considering the difference among the different channels. Our objective is to design a super-resolution model named Quaternion Attention Multi-scale Widening Network (QAMWN) for endoscopy images to address the above problem.

APPROACH

QAMWN contains a stacked Quaternion Attention Multi-Scale Widening Block (QAMWB), that composed of Multi-Scale Feature Widening Aggregation Module (MFWAM) and Quaternion Residual Channel Attention (QRCA). The MFWAM adopts multi-scale architecture with step-wise widening on feature channels for better feature extraction; and in QRCA, quaternion is introduced to construct Residual Channel Attention Mechanism, which obtains adaptively scales features by considering compact cross channel interactions in the hyper-complex domain.

MAIN RESULTS

To verify the efficacy of our method, it is performed on two public endoscopic datasets, CVC ClinicDB and Kvasir dataset. The experimental results show that our proposed method can achieve a better trade-off in model size and performance. More importantly, the proposed QAMWN outperforms previous state-of-the-art methods in both metrics and visualization.

SIGNIFICANCE

We propose a lightweight super-resolution network for endoscopy and achieves better performance with fewer parameters, which helps in clinical diagnosis of endoscopy.

Super-resolution 4D flow MRI to quantify aortic regurgitation using computational fluid dynamics and deep learning

Changes in cardiovascular hemodynamics are closely related to the development of aortic regurgitation (AR), a type of valvular heart disease. Metrics derived from blood flows are used to indicate AR onset and evaluate its severity. These metrics can be non-invasively obtained using four-dimensional (4D) flow magnetic resonance imaging (MRI), where accuracy is primarily dependent on spatial resolution. However, insufficient resolution often results from limitations in 4D flow MRI and complex aortic regurgitation hemodynamics. To address this, computational fluid dynamics simulations were transformed into synthetic 4D flow MRI data and used to train a variety of neural networks. These networks generated super-resolution, full-field phase images with an upsample factor of 4. Results showed decreased velocity error, high structural similarity scores, and improved learning capabilities from previous work. Further validation was performed on two sets of in vivo 4D flow MRI data and demonstrated success in de-noising flow images. This approach presents an opportunity to comprehensively analyse AR hemodynamics in a non-invasive manner.

Light Microscopy Technologies and the Plant Cytoskeleton

The cytoskeleton is a dynamic and diverse subcellular filament network, and as such microscopy is an essential technology to enable researchers to study and characterize these systems. Microscopy has a long history of observing the plant world not least as the subject where Robert Hooke coined the term "cell" in his publication Micrographia. From early observations of plant morphology to today's advanced super-resolution technologies, light microscopy is the indispensable tool for the plant cell biologist. In this mini review, we will discuss some of the major modalities used to examine the plant cytoskeleton and the theory behind them.

Expansion Microscopy of Plant Cells (PlantExM)

Expansion microscopy (ExM) achieves super-resolution imaging without the need for sophisticated super-resolution microscopy hardware through a combination of physical and optical magnification. Samples are fixed, stained, and embedded in a swellable gel. Following cross-linking of fluorophores to the gel matrix, the components of the sample are digested away and the gel expanded in water. Labeled objects which are too close to be resolved by diffraction-limited microscopy are moved far enough apart that these can now be resolved as individual objects on a standard confocal. Originally developed for animal cells and tissues, ExM for plants requires the additional consideration of cell wall digestion. Super-resolution can be limited in plants due to the size of cells, light scattering of tissues, and variations in refractive index. By removing the components which cause these limitations, ExM opens up the possibility of super-resolution at depth within plant tissues for the first time. Here we describe our method for PlantExM which is optimized for cytoskeleton resolution, which, when also coupled with compatible optical super-resolution technologies, can produce images of the plant cytoskeleton in unprecedented detail.

A Lightweight Low-Dose PET Image Super-resolution Reconstruction Method Based on Convolutional Neural Network

BACKGROUND

PET imaging is one of the most widely used neurological disease screening and diagnosis techniques.

AIMS

Since PET involves the radiation and tolerance of different people, the improvement that has always been focused on is to cut down radiation, in the meantime, ensuring that the generated images with low-dose tracer and generated images with standard-dose tracer have the same details of images.

METHODS

We propose a lightweight low-dose PET super-resolution network (SRPET-Net) based on a convolutional neural network. In this research, We propose a method for accurately recovering high-resolution (HR) PET images from low-resolution (LR) PET images. The network learns the details and structure of the image between low-dose PET images and standard-dose PET images and, afterward, reconstructs the PET image by the trained network model.

RESULTS

The experiments indicate that the SRPET-Net can achieve a superior peak signal-to-noise ratio (PSNR) and structural similarity index measurement (SSIM) values. Moreover, our method has less memory consumption and lower computational cost.

CONCLUSION

In our follow-up work, the technology can be applied to medical imaging in many different directions.

Deep Learning for Image Enhancement and Correction in Magnetic Resonance Imaging-State-of-the-Art and Challenges

Magnetic resonance imaging (MRI) provides excellent soft-tissue contrast for clinical diagnoses and research which underpin many recent breakthroughs in medicine and biology. The post-processing of reconstructed MR images is often automated for incorporation into MRI scanners by the manufacturers and increasingly plays a critical role in the final image quality for clinical reporting and interpretation. For image enhancement and correction, the post-processing steps include noise reduction, image artefact correction, and image resolution improvements. With the recent success of deep learning in many research fields, there is great potential to apply deep learning for MR image enhancement, and recent publications have demonstrated promising results. Motivated by the rapidly growing literature in this area, in this review paper, we provide a comprehensive overview of deep learning-based methods for post-processing MR images to enhance image quality and correct image artefacts. We aim to provide researchers in MRI or other research fields, including computer vision and image processing, a literature survey of deep learning approaches for MR image enhancement. We discuss the current limitations of the application of artificial intelligence in MRI and highlight possible directions for future developments. In the era of deep learning, we highlight the importance of a critical appraisal of the explanatory information provided and the generalizability of deep learning algorithms in medical imaging.

Channel attention generative adversarial network for super-resolution of glioma magnetic resonance image

BACKGROUND AND OBJECTIVE

Glioma is the most common primary craniocerebral tumor caused by the cancelation of glial cells in the brain and spinal cord, with a high incidence and cure rate. Magnetic resonance imaging (MRI) is a common technique for detecting and analyzing brain tumors. Due to improper hardware and operation, the obtained brain MRI images are low-resolution, making it difficult to detect and grade gliomas accurately. However, super-resolution reconstruction technology can improve the clarity of MRI images and help experts accurately detect and grade glioma.

METHODS

We propose a glioma magnetic resonance image super-resolution reconstruction method based on channel attention generative adversarial network (CGAN). First, we replace the base block of SRGAN with a residual dense block based on the channel attention mechanism. Second, we adopt a relative average discriminator to replace the discriminator in standard GAN. Finally, we add the mean squared error loss to the training, consisting of the mean squared error loss, the L1 norm loss, and the generator's adversarial loss to form the generator loss function.

RESULTS

On the Set5, Set14, Urban100, and glioma datasets, compared with the state-of-the-art algorithms, our proposed CGAN method has improved peak signal-to-noise ratio and structural similarity, and the reconstructed glioma images are more precise than other algorithms.

CONCLUSION

The experimental results show that our CGAN method has apparent improvements in objective evaluation indicators and subjective visual effects, indicating its effectiveness and superiority.

A super-resolution network using channel attention retention for pathology images

Image super-resolution (SR) significantly improves the quality of low-resolution images, and is widely used for image reconstruction in various fields. Although the existing SR methods have achieved distinguished results in objective metrics, most methods focus on real-world images and employ large and complex network structures, which are inefficient for medical diagnosis scenarios. To address the aforementioned issues, the distinction between pathology images and real-world images was investigated, and an SR Network with a wider and deeper attention module called Channel Attention Retention is proposed to obtain SR images with enhanced high-frequency features. This network captures contextual information within and across blocks via residual skips and balances the performance and efficiency by controlling the number of blocks. Meanwhile, a new linear loss was introduced to optimize the network. To evaluate the work and compare multiple SR works, a benchmark dataset bcSR was created, which forces a model training on wider and more critical regions. The results show that the proposed model outperforms state-of-the-art methods in both performance and efficiency, and the newly created dataset significantly improves the reconstruction quality of all compared models. Moreover, image classification experiments demonstrate that the suggested network improves the performance of downstream tasks in medical diagnosis scenarios. The proposed network and dataset provide effective priors for the SR task of pathology images, which significantly improves the diagnosis of relevant medical staff. The source code and the dataset are available on https://github.com/MoyangSensei/CARN-Pytorch.