Deep Learning for Ultrasound Localization Microscopy

Pruning Sparse Tensor Neural Networks Enables Deep Learning for 3D Ultrasound Localization Microscopy

Ultrasound Localization Microscopy (ULM) is a non-invasive technique that allows for the imaging of micro-vessels in vivo, at depth and with a resolution on the order of ten microns. ULM is based on the sub-resolution localization of individual microbubbles injected in the bloodstream. Mapping the whole angioarchitecture requires the accumulation of microbubbles trajectories from thousands of frames, typically acquired over a few minutes. ULM acquisition times can be reduced by increasing the microbubble concentration, but requires more advanced algorithms to detect them individually. Several deep learning approaches have been proposed for this task, but they remain limited to 2D imaging, in part due to the associated large memory requirements. Herein, we propose the use of sparse tensor neural networks to enable deep learning-based 3D ULM by improving memory scalability with increased dimensionality. We study several approaches to efficiently convert ultrasound data into a sparse format and study the impact of the associated loss of information. When applied in 2D, the sparse formulation reduces the memory requirements by a factor 2 at the cost of a small reduction of performance when compared against dense networks. In 3D, the proposed approach reduces memory requirements by two order of magnitude while largely outperforming conventional ULM in high concentration settings. We show that Sparse Tensor Neural Networks in 3D ULM allow for the same benefits as dense deep learning based method in 2D ULM i.e. the use of higher concentration in silico and reduced acquisition time.

Vasomics of the liver

Chronic liver disease is a cluster of disorders associated with complex haemodynamic alterations, which is characterised by structural and functional disruptions of the intrahepatic and extrahepatic vasculature. 'Vasomics' is an emerging omics discipline that comprehensively analyses and models the vascular system by integrating pathophysiology of disease, biomechanics, medical imaging, computational science and artificial intelligence. Vasomics is further typified by its multidimensional, multiscale and high-throughput nature, which depends on the rapid and robust extraction of well-defined vascular phenotypes with clear clinical and/or biological interpretability. By leveraging multimodality medical imaging techniques, vascular functional assessments, pathological image evaluation, and related computational methods, integrated vasomics provides a deeper understanding of the associations between the vascular system and disease. This in turn reveals the crucial role of the vascular system in disease occurrence, progression and treatment responses, thereby supporting precision medicine approaches. Pathological vascular features have already demonstrated their key role in different clinical scenarios. Despite this, vasomics is yet to be widely recognised. Therefore, we furnished a comprehensive definition of vasomics providing a classification of existing hepatic vascular phenotypes into the following categories: anatomical, biomechanical, biochemical, pathophysiological and composite.

Improved Microbubble Tracking for Super-Resolution Ultrasound Localization Microscopy using a Bi-Directional Long Short-term Memory Neural Network

Ultrasound localization microscopy (ULM) enabled high-accuracy measurements of microvessel flow beyond the resolution limit of conventional ultrasound imaging by utilizing contrast microbubbles (MBs) as point targets. Robust tracking of MBs is an essential task for fast and high-quality ULM image reconstruction. Existing MB tracking methods suffer from challenging imaging scenarios such as high-density MB distributions, fast blood flow, and complex flow dynamics. Here we present a deep learning-based MB pairing and tracking method based on a bi-directional long short-term memory neural network for ULM. The proposed method integrates multiparametric MB characteristics to facilitate more robust and accurate MB pairing and tracking. The method was validated on a simulation data set, a tissue-mimicking flow phantom, and in vivo on a mouse and rat brain.

A Tracking Prior to Localization Workflow for Ultrasound Localization Microscopy

Ultrasound Localization Microscopy (ULM) has proven effective in resolving microvascular structures and local mean velocities at sub-diffraction-limited scales, offering high-resolution imaging capabilities. Dynamic ULM (DULM) enables the creation of angiography or velocity movies throughout cardiac cycles. Currently, these techniques rely on a Localization-and-Tracking (LAT) workflow consisting in detecting microbubbles (MB) in the frames before pairing them to generate tracks. While conventional LAT methods perform well at low concentrations, they suffer from longer acquisition times and degraded localization and tracking accuracy at higher concentrations, leading to biased angiogram reconstruction and velocity estimation. In this study, we propose a novel approach to address these challenges by reversing the current workflow. The proposed method, Tracking-and-Localization (TAL), relies on first tracking the MB and then performing localization. Through comprehensive benchmarking using both in silico and in vivo experiments and employing various metrics to quantify ULM angiography and velocity maps, we demonstrate that the TAL method consistently outperforms the reference LAT workflow. Moreover, when applied to DULM, TAL successfully extracts velocity variations along the cardiac cycle with improved repeatability. The findings of this work highlight the effectiveness of the TAL approach in overcoming the limitations of conventional LAT methods, providing enhanced ULM angiography and velocity imaging.

Ultrasound localization microscopy in the diagnosis of breast tumors and prediction of relevant histologic biomarkers associated with prognosis in humans: the protocol for a prospective, multicenter study

BACKGROUND

Benign and malignant breast tumors differ in their microvasculature morphology and distribution. Histologic biomarkers of malignant breast tumors are also correlated with the microvasculature. There is a lack of imaging technology for evaluating the microvasculature. Ultrasound localization microscopy (ULM) can provide detailed microvascular architecture at super-resolution. The objective of this trial is to explore the role of ULM in distinguishing benign from malignant breast tumors and to explore the correlations between ULM qualitative and quantitative parameters and histologic biomarkers in malignant breast tumors.

METHODS/DESIGN

This prospective and multicenter study will include 83 patients with breast tumors that will undergo ULM. 55 patients will be assigned to the malignant group, and 28 patients will be assigned to the benign group. The primary outcome is the differences in the qualitative parameters (microvasculature morphology, distribution, and flow direction) between benign and malignant breast tumors on ULM. Secondary outcomes include (1) differences in the quantitative parameters (microvasculature density, tortuosity, diameter, and flow velocity) between benign and malignant breast tumors based on ULM; (2) diagnostic performance of the qualitative parameters in distinguishing benign and malignant breast tumors; (3) diagnostic performance of the quantitative parameters in distinguishing benign and malignant breast tumors; (4) relationships between the qualitative parameters and histologic biomarkers in malignant breast tumors; (5) relationships between the quantitative parameters and histologic biomarkers in malignant breast tumors; and (6) the evaluation of inter-reader and intra-reader reproducibility.

DISCUSSION

Detecting vascularity in breast tumors is of great significance to differentiate benign from malignant tumors and to predict histologic biomarkers. These histologic biomarkers, such as ER, PR, HER2 and Ki67, are closely related to prognosis evaluation. This trial will provide maximum information about the microvasculature of breast tumors and thereby will help with the formulation of subsequent differential diagnosis and the prediction of histologic biomarkers.

TRIAL REGISTRATION NUMBER/DATE

Chinese Clinical Trial Registry ChiCTR2100048361/6th/July/2021. This study is a part of that clinical trial.

Microbubble tracking based on partial smoothing-based adaptive generalized labelled Multi-Bernoulli filter for super-resolution imaging

Super-resolution ultrasound (SRUS) can image the vasculature at microscopic resolution according to microbubble (MB) localization, with velocity vector maps obtained based on MB tracking information. High MB concentrations can reduce the acquisition time of SRUS imaging, however adjacent and intersecting vessels are difficult to distinguish, thus decreasing resolution. Low acquisition frame rates affect the precision of flow velocity estimation. This study proposes a partial smoothing-based adaptive generalized labeled multi-Bernoulli filter (SAGLMB) to precisely track the MB motion at different flow velocities. SAGLMB employs a generalized labelled multi-Bernoulli filter (GLMB) for MB trajectory allocation to separate adjacent and intersecting vessels. Furthermore, the nonlinear motion of MB was predicted by an unscented Kalman filter, and a cardinalized probability hypothesis density filter was applied to suppress clutter interference. Finally, the trajectories were smoothed by unscented Rauch-Tung-Striebel to improve the resolution of the SRUS image. The simulation results demonstrate that SAGLMB outperforms the conventional bipartite graph-based tracking at high MB concentrations, achieving at least an 8.55 % improvement in the correctly paired precision, with 3 times increase in the structural similarity index measure. Moreover, SAGLMB can obtain more precise flow velocity estimations with a 4 times improvement than the conventional method. The SRUS results of rabbit kidney show that the proposed method significantly improves resolution of adjacent and intersecting vessels at higher MB concentrations and maintains this performance as the acquisition frame rate decreases. Furthermore, the rat brain microvascular network was reconstructed with 9.21 μm (λ/11.1) resolution. Therefore, SAGLMB can achieve robust SRUS imaging at high concentrations and low acquisition frame rates.

Super-Resolution Ultrasound Imaging for Analysis of Microbubbles Cluster by Acoustic Vortex Tweezers

Using acoustic vortex tweezers (AVTs) to spatially accumulate microbubbles (MBs) shows promise for enhancing drug delivery efficiency and reducing off-target effects. The strong echogenicity of accumulated MBs also improves diagnostics via conventional ultrasound (US) B-mode imaging. However, the annular high-pressure distribution of AVT inhibits MBs inflow at the inlet, reducing MBs collection. The spatial resolution of US B-mode imaging further limits theranostic applications of AVT-mediated MBs accumulation. To address these challenges, we integrated an AVT waveform with volumetric super-resolution imaging (VSRI) to monitor the dynamic growth of MBs cluster during accumulation. We used a 5-MHz 2-D array transducer for VSRI, employing plane wave pulses interleaved with accumulating pulses to retain MBs at a flow rate of 0.023-0.047 mL/s in a 3-mm vessel phantom. An asymmetrical AVT waveform (AVT ) was produced by modulating the pressure at the MBs inlet compared to the outlet. The effectiveness was validated in rat cerebral vessels for real-time volumetric tracking of MBs clusters. Microscopy observations showed that AVT could quickly gather flowing MBs into cluster without repelling them at a flow rate of 0.023 mL/s. Statistical results indicated that microscopic data correlated better with VSRI than with B-mode images, suggesting VSRI suffices to detect the dynamics of AVT -actuated MBs accumulation in real-time. Additionally, VSRI detected a significant increase in MBs cluster size over time during AVT in the superior sagittal sinus (SSS) of the rat brain. These findings demonstrate that the proposed strategy can accumulate the flowing MBs at a desired location and simultaneously observe this phenomenon.

Deep Learning in Ultrasound Localization Microscopy: Applications and Perspectives

Ultrasound localization microscopy (ULM) is a novel super-resolution imaging technique that can image the vasculature in vivo at depth with resolution far beyond the conventional limit of diffraction. By relying on the localization and tracking of clinically approved microbubbles injected in the blood stream, ULM can provide not only anatomical visualization but also hemodynamic quantification of the microvasculature. Several deep learning approaches have been proposed to address challenges in ULM including denoising, improving microbubble localization, estimating blood flow velocity, or performing aberration correction. Proposed deep learning methods often outperform their conventional counterparts by improving image quality and reducing processing time. In addition, their robustness to high concentrations of microbubbles can lead to reduced acquisition times in ULM, addressing a major hindrance to ULM clinical application. Herein, we propose a comprehensive review of the diversity of deep learning applications in ULM focusing on approaches assuming a sparse microbubble distribution. We first provide an overview of how existing studies vary in the constitution of their datasets or in the tasks targeted by the deep learning model. We also take a deeper look into the numerous approaches that have been proposed to improve the localization of microbubbles since they differ highly in their formulation of the optimization problem, their evaluation, or their network architectures. We finally discuss the current limitations and challenges of these methods, as well as the promises and potential of deep learning for ULM in the future.

ULM-MbCNRT: In Vivo Ultrafast Ultrasound Localization Microscopy by Combining Multibranch CNN and Recursive Transformer

Ultrasound localization microscopy (ULM) overcomes the acoustic diffraction limit by localizing tiny microbubbles (MBs), thus enabling the microvascular to be rendered at subwavelength resolution. Nevertheless, to obtain such superior spatial resolution, it is necessary to spend tens of seconds gathering numerous ultrasound (US) frames to accumulate the MB events required, resulting in ULM imaging still suffering from tradeoffs between imaging quality, data acquisition time, and data processing speed. In this article, we present a new deep learning (DL) framework combining multibranch convolutional neural network (CNN) and recursive transformer (RT), termed ULM-MbCNRT, that is capable of reconstructing a super-resolution (SR) image directly from a temporal mean low-resolution image generated by averaging much fewer raw US frames, i.e., implement an ultrafast ULM imaging. To evaluate the performance of ULM-MbCNRT, a series of numerical simulations and in vivo experiments are carried out. Numerical simulation results indicate that ULM-MbCNRT achieves high-quality ULM imaging with ~10-fold reduction in data acquisition time and ~130-fold reduction in computation time compared to the previous DL method (e.g., the modified subpixel CNN, ULM-mSPCN). For the in vivo experiments, when comparing to the ULM-mSPCN, ULM-MbCNRT allows ~37-fold reduction in data acquisition time (~0.8 s) and ~2134-fold reduction in computation time (~0.87 s) without sacrificing spatial resolution. It implies that ultrafast ULM imaging holds promise for observing rapid biological activity in vivo, potentially improving the diagnosis and monitoring of clinical conditions.

Deep Power-Aware Tunable Weighting for Ultrasound Microvascular Imaging

Ultrasound microvascular imaging (UMI), including ultrafast power Doppler imaging (uPDI) and ultrasound localization microscopy (ULM), obtains blood flow information through plane wave (PW) transmissions at high frame rates. However, low signal-to-noise ratio (SNR) of PWs causes low image quality. Adaptive beamformers have been proposed to suppress noise energy to achieve higher image quality accompanied by increasing computational complexity. Deep learning (DL) leverages powerful hardware capabilities to enable rapid implementation of noise suppression at the cost of flexibility. To enhance the applicability of DL-based methods, in this work, we propose a deep power-aware tunable (DPT) weighting (i.e., postfilter) for delay-and-sum (DAS) beamforming to improve UMI by enhancing PW images. The model, called Yformer, is a hybrid structure combining convolution and Transformer. With the DAS beamformed and compounded envelope image as input, Yformer can estimate both noise power and signal power. Furthermore, we utilize the obtained powers to compute pixel-wise weights by introducing a tunable noise control factor (NCF), which is tailored for improving the quality of different UMI applications. In vivo experiments on the rat brain demonstrate that Yformer can accurately estimate the powers of noise and signal with the structural similarity index measure (SSIM) higher than 0.95. The performance of the DPT weighting is comparable to that of superior adaptive beamformer in uPDI with low computational cost. The DPT weighting was then applied to four different datasets of ULM, including public simulation, public rat brain, private rat brain, and private rat liver datasets, showing excellent generalizability using the model trained by the private rat brain dataset only. In particular, our method indirectly improves the resolution of liver ULM from 25.24 to m by highlighting small vessels. In addition, the DPT weighting exhibits more details of blood vessels with faster processing, which has the potential to facilitate the clinical applications of high-quality UMI.

VoxelMorph-Based Deep Learning Motion Correction for Ultrasound Localization Microscopy of Spinal Cord

Accurate assessment of spinal cord vasculature is important for the urgent diagnosis of injury and subsequent treatment. Ultrasound localization microscopy (ULM) offers super-resolution imaging of microvasculature by localizing and tracking individual microbubbles (MBs) across multiple frames. However, a long data acquisition often involves significant motion artifacts caused by breathing and heartbeat, which further impairs the resolution of ULM. This effect is particularly pronounced in spinal cord imaging due to respiratory movement. We propose a VoxelMorph-based deep learning (DL) motion correction method to enhance the ULM performance in spinal cord imaging. Simulations were conducted to demonstrate the motion estimation accuracy of the proposed method, achieving a mean absolute error of m. Results from in vivo experiments show that the proposed method efficiently compensates for rigid and nonrigid motion, providing improved resolution with smaller vascular diameters and enhanced microvessel reconstruction after motion correction. Nonrigid deformation fields with varying displacement magnitudes were applied to in vivo data for assessing the robustness of the algorithm.

Ultrasound Localization Microscopy Precision of Clinical 3-D Ultrasound Systems

Ultrasound localization microscopy (ULM) is becoming well established in preclinical applications. For its translation into clinical practice, the localization precision achievable with commercial ultrasound (US) scanners is crucial-especially with volume imaging, which is essential for dealing with out-of-plane motion. Here, we propose an easy-to-perform method to estimate the localization precision of 3-D US scanners. With this method, we evaluated imaging sequences of the Philips Epiq 7 US device using the X5-1 and the XL14-3 matrix transducers and also tested different localization methods. For the X5-1 transducer, the best lateral, elevational, and axial precision was 109, 95, and m for one contrast mode, and 29, 22, and m for the other. The higher frequency XL14-3 transducer yielded precisions of 17, 38, and m using the harmonic imaging mode. Although the center of mass was the most robust localization method also often providing the best precision, the localization method has only a minor influence on the localization precision compared to the impact by the imaging sequence and transducer. The results show that with one of the imaging modes of the X5-1 transducer, precisions comparable to the XL14-3 transducer can be achieved. However, due to localization precisions worse than m, reconstruction of the microvasculature at the capillary level will not be possible. These results show the importance of evaluating the localization precision of imaging sequences from different US transducers or scanners in all directions before using them for in vivo measurements.

Efficient Microbubble Trajectory Tracking in Ultrasound Localization Microscopy Using a Gated Recurrent Unit-Based Multitasking Temporal Neural Network

Ultrasound localization microscopy (ULM), an emerging medical imaging technique, effectively resolves the classical tradeoff between resolution and penetration inherent in traditional ultrasound imaging, opening up new avenues for noninvasive observation of the microvascular system. However, traditional microbubble tracking methods encounter various practical challenges. These methods typically entail multiple processing stages, including intricate steps such as pairwise correlation and trajectory optimization, rendering real-time applications unfeasible. Furthermore, existing deep learning-based tracking techniques neglect the temporal aspects of microbubble motion, leading to ineffective modeling of their dynamic behavior. To address these limitations, this study introduces a novel approach called the gated recurrent unit-based multitasking temporal neural network (GRU-MT). GRU-MT is designed to simultaneously handle microbubble trajectory tracking and trajectory optimization tasks. In addition, we enhance the nonlinear motion model initially proposed by Piepenbrock et al. to better encapsulate the nonlinear motion characteristics of microbubbles, thereby improving trajectory tracking accuracy. In this study, we perform a series of experiments involving network layer replacements to systematically evaluate the performance of various temporal neural networks, including recurrent neural network (RNN), long short-term memory network (LSTM), GRU, Transformer, and its bidirectional counterparts, on the microbubble trajectory tracking task. Concurrently, the proposed method undergoes qualitative and quantitative comparisons with traditional microbubble tracking techniques. The experimental results demonstrate that GRU-MT exhibits superior nonlinear modeling capabilities and robustness, both in simulation and in vivo dataset. In addition, it achieves reduced trajectory tracking errors in shorter time intervals, underscoring its potential for efficient microbubble trajectory tracking. The model code is open-sourced at https://github.com/zyt-Lib/GRU-MT.

Artificial intelligence diagnosis of patent foramen ovale in contrast transthoracic echocardiography

Artificial intelligence (AI) is rarely directly used in patent foramen ovale (PFO) diagnosis. In this study, an AI model was developed to detect the presence of PFO automatically in both contrast transthoracic echocardiography (cTTE) images and videos. The whole intelligent diagnosis neural network framework consists of two functional modules of image segmentation (Unet, n = 1866) and image classification (ResNet 101, n = 9152). Finally, another test databases, including 20 cTTE videos (4609 cTTE images), was used to compare the RLS classification model accuracy between AI model and different levels of physicians. The Dice similarity coefficient of left chamber segmentation model of cTTE images was 91.41%, the accuracy of PFO-RLS classification model of cTTE images was 83.55%, the accuracy of PFO-RLS classification model of cTTE videos was 90%. Besides, the AI diagnosis time was significantly shorter than doctors (at only 1.3 s).

Super-resolution ultrasound imaging reveals temporal cerebrovascular changes with disease progression in female 5×FAD mouse model of Alzheimer's disease: correlation with pathological impairments

BACKGROUND

Vascular dysfunction is closely associated with the progression of Alzheimer's disease (AD). A critical research gap exists that no studies have explored the in vivo temporal changes of cerebrovascular alterations with AD progression in mouse models, encompassing both structure and flow dynamics at micron-scale resolution across the early, middle, and late stages of the disease.

METHODS

In this study, ultrasound localisation microscopy (ULM) was applied to image the cerebrovascular alterations of the transgenic female 5×FAD mouse model across different stages of disease progression: early (4 months), moderate (7 months), and late (12 months). Age-matched non-transgenic (non-Tg) littermates were used as controls. Immunohistology examinations were performed to evaluate the influence of disease progression on the β-amyloid (Aβ) load and microvascular alterations, including morphological changes and the blood-brain barrier (BBB) leakage.

FINDINGS

Our findings revealed a significant decline in both vascular density and flow velocity in the retrosplenial cortex of 5×FAD mice at an early stage, which subsequently became more pronounced in the visual cortex and hippocampus as the disease progressed. Additionally, we observed a reduction in vascular length preceding diminished flow velocities in cortical penetrating arterioles during the early stages. The quantification of vascular metrics derived from ULM imaging showed significant correlations with those obtained from vascular histological images. Immunofluorescence staining identified early vascular abnormalities in the retrosplenial cortex. As the disease advanced, there was an exacerbation of Aβ accumulation and BBB disruption in a regionally variable manner. The vascular changes observed through ULM imaging exhibited a negative correlation with amyloid load and were associated with the compromise of the BBB integrity.

INTERPRETATION

Through high-resolution, in vivo imaging of cerebrovasculature, this study reveals significant spatiotemporal dysfunction in cerebrovascular dynamics accompanying disease progression in a mouse model of AD, enhancing our understanding of its pathophysiology.

FUNDING

This study is supported by grants from National Key Research and Development Program of China (2020YFA0908800), National Natural Science Foundation of China (12074269, 82272014, 82327804, 62071310), Shenzhen Basic Science Research (20220808185138001, JCYJ20220818095612027, JCYJ20210324093006017), STI 2030-Major Projects (2021ZD0200500) and Guangdong Natural Science Foundation (2024A1515012591, 2024A1515011342).

RF-ULM: Ultrasound Localization Microscopy Learned From Radio-Frequency Wavefronts

In Ultrasound Localization Microscopy (ULM), achieving high-resolution images relies on the precise localization of contrast agent particles across a series of beamformed frames. However, our study uncovers an enormous potential: The process of delay-and-sum beamforming leads to an irreversible reduction of Radio-Frequency (RF) channel data, while its implications for localization remain largely unexplored. The rich contextual information embedded within RF wavefronts, including their hyperbolic shape and phase, offers great promise for guiding Deep Neural Networks (DNNs) in challenging localization scenarios. To fully exploit this data, we propose to directly localize scatterers in RF channel data. Our approach involves a custom super-resolution DNN using learned feature channel shuffling, non-maximum suppression, and a semi-global convolutional block for reliable and accurate wavefront localization. Additionally, we introduce a geometric point transformation that facilitates seamless mapping to the B-mode coordinate space. To understand the impact of beamforming on ULM, we validate the effectiveness of our method by conducting an extensive comparison with State-Of-The-Art (SOTA) techniques. We present the inaugural in vivo results from a wavefront-localizing DNN, highlighting its real-world practicality. Our findings show that RF-ULM bridges the domain shift between synthetic and real datasets, offering a considerable advantage in terms of precision and complexity. To enable the broader research community to benefit from our findings, our code and the associated SOTA methods are made available at https://github.com/hahnec/rf-ulm.

In vivo ultrasound localization microscopy for high-density microbubbles

Ultrasound Localization Microscopy (ULM) surpasses the constraints imposed by acoustic diffraction, achieving sub-wavelength resolution visualization of microvasculature through the precise localization of minute microbubbles (MBs). Nonetheless, the analysis of densely populated regions with overlapping MB point spread responses introduces significant localization errors, limiting the use of technique to low-concentration conditions. This raises a trade-off issue between localization efficiency and MB density. In this work, we present a new deep learning framework that combines Transformer and U-Net architectures, termed ULM-TransUNet. As a non-linear model, it is able to learn the complex data patterns of overlapping MBs in dense conditions for accurate localization. To evaluate the performance of ULM-TransUNet, a series of numerical simulations and in vivo experiments are carried out. Numerical simulation results indicate that ULM-TransUNet achieves high-quality ULM imaging, with improvements of 21.93 % in detection rate, 17.36 % in detection precision, and 20.53 % in detection sensitivity, compared to previous state-of-the-art deep learning (DL) method (e.g., ULM-UNet). For the in vivo experiments, ULM-TransUNet achieves the highest spatial resolution (9.4 μm) and rapid inference speed (26.04 ms/frame). Furthermore, it consistently detects more small vessels and resolves closely spaced vessels more effectively. The outcomes of this work imply that ULM-TransUNet can potentially enhance the microvascular imaging performance on high-density MB conditions.

Context-aware deep learning enables high-efficacy localization of high concentration microbubbles for super-resolution ultrasound localization microscopy

Ultrasound localization microscopy (ULM) enables deep tissue microvascular imaging by localizing and tracking intravenously injected microbubbles circulating in the bloodstream. However, conventional localization techniques require spatially isolated microbubbles, resulting in prolonged imaging time to obtain detailed microvascular maps. Here, we introduce LOcalization with Context Awareness (LOCA)-ULM, a deep learning-based microbubble simulation and localization pipeline designed to enhance localization performance in high microbubble concentrations. In silico, LOCA-ULM enhanced microbubble detection accuracy to 97.8% and reduced the missing rate to 23.8%, outperforming conventional and deep learning-based localization methods up to 17.4% in accuracy and 37.6% in missing rate reduction. In in vivo rat brain imaging, LOCA-ULM revealed dense cerebrovascular networks and spatially adjacent microvessels undetected by conventional ULM. We further demonstrate the superior localization performance of LOCA-ULM in functional ULM (fULM) where LOCA-ULM significantly increased the functional imaging sensitivity of fULM to hemodynamic responses invoked by whisker stimulations in the rat brain.

Improving sub-pixel accuracy in ultrasound localization microscopy using supervised and self-supervised deep learning

With a spatial resolution of tens of microns, ultrasound localization microscopy (ULM) reconstructs microvascular structures and measures intravascular flows by tracking microbubbles (1-5 μm) in contrast enhanced ultrasound (CEUS) images. Since the size of CEUS bubble traces, e.g. 0.5-1 mm for ultrasound with a wavelength λ = 280 μm, is typically two orders of magnitude larger than the bubble diameter, accurately localizing microbubbles in noisy CEUS data is vital to the fidelity of the ULM results. In this paper, we introduce a residual learning based supervised super-resolution blind deconvolution network (SupBD-net), and a new loss function for a self-supervised blind deconvolution network (SelfBD-net), for detecting bubble centers at a spatial resolution finer than λ/10. Our ultimate purpose is to improve the ability to distinguish closely located microvessels and the accuracy of the velocity profile measurements in macrovessels. Using realistic synthetic data, the performance of these methods is calibrated and compared against several recently introduced deep learning and blind deconvolution techniques. For bubble detection, errors in bubble center location increase with the trace size, noise level, and bubble concentration. For all cases, SupBD-net yields the least error, keeping it below 0.1 λ. For unknown bubble trace morphology, where all the supervised learning methods fail, SelfBD-net can still maintain an error of less than 0.15 λ. SupBD-net also outperforms the other methods in separating closely located bubbles and parallel microvessels. In macrovessels, SupBD-net maintains the least errors in the vessel radius and velocity profile after introducing a procedure that corrects for terminated tracks caused by overlapping traces. Application of these methods is demonstrated by mapping the cerebral microvasculature of a neonatal pig, where neighboring microvessels separated by 0.15 λ can be readily distinguished by SupBD-net and SelfBD-net, but not by the other techniques. Hence, the newly proposed residual learning based methods improve the spatial resolution and accuracy of ULM in micro- and macro-vessels.

Competitive Swarm Optimized SVD Clutter Filtering for Ultrafast Power Doppler Imaging

Ultrafast power Doppler imaging (uPDI) can significantly increase the sensitivity of resolving small vascular paths in ultrasound. While clutter filtering is a fundamental and essential method to realize uPDI, it commonly uses singular value decomposition (SVD) to suppress clutter signals and noise. However, current SVD-based clutter filters using two cutoffs cannot ensure sufficient separation of tissue, blood, and noise in uPDI. This article proposes a new competitive swarm-optimized SVD clutter filter to improve the quality of uPDI. Specifically, without using two cutoffs, such a new filter introduces competitive swarm optimization (CSO) to search for the counterparts of blood signals in each singular value. We validate the CSO-SVD clutter filter on public in vivo datasets. The experimental results demonstrate that our method can achieve higher contrast-to-noise ratio (CNR), signal-to-noise ratio (SNR), and blood-to-clutter ratio (BCR) than the state-of-the-art SVD-based clutter filters, showing a better balance between suppressing clutter signals and preserving blood signals. Particularly, our CSO-SVD clutter filter improves CNR by 0.99 ± 0.08 dB, SNR by 0.79 ± 0.08 dB, and BCR by 1.95 ± 0.03 dB when comparing a spatial-similarity-based SVD clutter filter in the in vivo dataset of rat brain bolus.

Deep learning for fast super-resolution ultrasound microvessel imaging

Objective. Ultrasound localization microscopy (ULM) enables microvascular reconstruction by localizing microbubbles (MBs). Although ULM can obtain microvascular images that are beyond the ultimate resolution of the ultrasound (US) diffraction limit, it requires long data processing time, and the imaging accuracy is susceptible to the density of MBs. Deep learning (DL)-based ULM is proposed to alleviate these limitations, which simulated MBs at low-resolution and mapped them to coordinates at high-resolution by centroid localization. However, traditional DL-based ULMs are imprecise and computationally complex. Also, the performance of DL is highly dependent on the training datasets, which are difficult to realistically simulate.Approach. A novel architecture called adaptive matching network (AM-Net) and a dataset generation method named multi-mapping (MMP) was proposed to overcome the above challenges. The imaging performance and processing time of the AM-Net have been assessed by simulation andin vivoexperiments.Main results. Simulation results show that at high density (20 MBs/frame), when compared to other DL-based ULM, AM-Net achieves higher localization accuracy in the lateral/axial direction.In vivoexperiment results show that the AM-Net can reconstruct ∼24.3μm diameter micro-vessels and separate two ∼28.3μm diameter micro-vessels. Furthermore, when processing a 128 × 128 pixels image in simulation experiments and an 896 × 1280 pixels imagein vivoexperiment, the processing time of AM-Net is ∼13 s and ∼33 s, respectively, which are 0.3-0.4 orders of magnitude faster than other DL-based ULM.Significance. We proposes a promising solution for ULM with low computing costs and high imaging performance.

Three-dimensional super resolution ultrasound imaging with a multi-frequency hemispherical phased array

BACKGROUND

High resolution imaging of the microvasculature plays an important role in both diagnostic and therapeutic applications in the brain. However, ultrasound pulse-echo sonography imaging the brain vasculatures has been limited to narrow acoustic windows and low frequencies due to the distortion of the skull bone, which sacrifices axial resolution since it is pulse length dependent.

PURPOSE

To overcome the detect limit, a large aperture 256-module sparse hemispherical transmit/receive array was used to visualize the acoustic emissions of ultrasound-vaporized lipid-coated decafluorobutane nanodroplets flowing through tube phantoms and within rabbit cerebral vasculature in vivo via passive acoustic mapping and super resolution techniques.

METHODS

Nanodroplets were vaporized with 55 kHz burst-mode ultrasound (burst length = 145 μs, burst repetition frequency = 9-45 Hz, peak negative acoustic pressure = 0.10-0.22 MPa), which propagates through overlying tissues well without suffering from severe distortions. The resulting emissions were received at a higher frequency (612 or 1224 kHz subarray) to improve the resulting spatial resolution during passive beamforming. Normal resolution three-dimensional images were formed using a delay, sum, and integrate beamforming algorithm, and super-resolved images were extracted via Gaussian fitting of the estimated point-spread-function to the normal resolution data.

RESULTS

With super resolution techniques, the mean lateral (axial) full-width-at-half-maximum image intensity was 16 ± 3 (32 ± 6) μm, and 7 ± 1 (15 ± 2) μm corresponding to ∼1/67 of the normal resolution at 612 and 1224 kHz, respectively. The mean positional uncertainties were ∼1/350 (lateral) and ∼1/180 (axial) of the receive wavelength in water. In addition, a temporal correlation between nanodroplet vaporization and the transmit waveform shape was observed, which may provide the opportunity to enhance the signal-to-noise ratio in future studies.

CONCLUSIONS

Here, we demonstrate the feasibility of vaporizing nanodroplets via low frequency ultrasound and simultaneously performing spatial mapping via passive beamforming at higher frequencies to improve the resulting spatial resolution of super resolution imaging techniques. This method may enable complete four-dimensional vascular mapping in organs where a hemispherical array could be positioned to surround the target, such as the brain, breast, or testicles.

Deep learning for fast denoising filtering in ultrasound localization microscopy

Objective.Addition of a denoising filter step in ultrasound localization microscopy (ULM) has been shown to effectively reduce the error localizations of microbubbles (MBs) and achieve resolution improvement for super-resolution ultrasound (SR-US) imaging. However, previous image-denoising methods (e.g. block-matching 3D, BM3D) requires long data processing times, making ULM only able to be processed offline. This work introduces a new way to reduce data processing time through deep learning.Approach.In this study, we propose deep learning (DL) denoising based on contrastive semi-supervised network (CS-Net). The neural network is mainly trained with simulated MBs data to extract MB signals from noise. And the performances of CS-Net denoising are evaluated in bothin vitroflow phantom experiment andin vivoexperiment of New Zealand rabbit tumor.Main results.Forin vitroflow phantom experiment, the signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) of single microbubble image are 26.91 dB and 4.01 dB, repectively. Forin vivoanimal experiment , the SNR and CNR were 12.29 dB and 6.06 dB. In addition, single microvessel of 24μm and two microvessels separated by 46μm could be clearly displayed. Most importantly,, the CS-Net denoising speeds forin vitroandin vivoexperiments were 0.041 s frame-1and 0.062 s frame-1, respectively.Significance.DL denoising based on CS-Net can improve the resolution of SR-US as well as reducing denoising time, thereby making further contributions to the clinical real-time imaging of ULM.

Nanodroplet-Based Super-Resolution Ultrasound Localization Microscopy

Over the past decade, super-resolution ultrasound localization microscopy (SR-ULM) has revolutionized ultrasound imaging with its capability to resolve the microvascular structures below the ultrasound diffraction limit. The introduction of this imaging technique enables the visualization, quantification, and characterization of tissue microvasculature. The early implementations of SR-ULM utilize microbubbles (MBs) that require a long image acquisition time due to the requirement of capturing sparsely isolated microbubble signals. The next-generation SR-ULM employs nanodroplets that have the potential to significantly reduce the image acquisition time without sacrificing the resolution. This review discusses various nanodroplet-based ultrasound localization microscopy techniques and their corresponding imaging mechanisms. A summary is given on the preclinical applications of SR-ULM with nanodroplets, and the challenges in the clinical translation of nanodroplet-based SR-ULM are presented while discussing the future perspectives. In conclusion, ultrasound localization microscopy is a promising microvasculature imaging technology that can provide new diagnostic and prognostic information for a wide range of pathologies, such as cancer, heart conditions, and autoimmune diseases, and enable personalized treatment monitoring at a microlevel.

Localization Free Super-Resolution Microbubble Velocimetry Using a Long Short-Term Memory Neural Network

Ultrasound localization microscopy is a super-resolution imaging technique that exploits the unique characteristics of contrast microbubbles to side-step the fundamental trade-off between imaging resolution and penetration depth. However, the conventional reconstruction technique is confined to low microbubble concentrations to avoid localization and tracking errors. Several research groups have introduced sparsity- and deep learning-based approaches to overcome this constraint to extract useful vascular structural information from overlapping microbubble signals, but these solutions have not been demonstrated to produce blood flow velocity maps of the microcirculation. Here, we introduce Deep-SMV, a localization free super-resolution microbubble velocimetry technique, based on a long short-term memory neural network, that provides high imaging speed and robustness to high microbubble concentrations, and directly outputs blood velocity measurements at a super-resolution. Deep-SMV is trained efficiently using microbubble flow simulation on real in vivo vascular data and demonstrates real-time velocity map reconstruction suitable for functional vascular imaging and pulsatility mapping at super-resolution. The technique is successfully applied to a wide variety of imaging scenarios, include flow channel phantoms, chicken embryo chorioallantoic membranes, and mouse brain imaging. An implementation of Deep-SMV is openly available at https://github.com/chenxiptz/SR_microvessel_velocimetry, with two pre-trained models available at https://doi.org/10.7910/DVN/SECUFD.

Super-resolution ultrasound microvascular imaging: Is it ready for clinical use?

The field of super-resolution ultrasound microvascular imaging has been rapidly growing over the past decade. By leveraging contrast microbubbles as point targets for localization and tracking, super-resolution ultrasound pinpoints the location of microvessels and measures their blood flow velocity. Super-resolution ultrasound is the first in vivo imaging modality that can image micron-scale vessels at a clinically relevant imaging depth without tissue destruction. These unique capabilities of super-resolution ultrasound provide structural (vessel morphology) and functional (vessel blood flow) assessments of tissue microvasculature on a global and local scale, which opens new doors for many enticing preclinical and clinical applications that benefit from microvascular biomarkers. The goal of this short review is to provide an update on recent advancements in super-resolution ultrasound imaging, with a focus on summarizing existing applications and discussing the prospects of translating super-resolution imaging to clinical practice and research. In this review, we also provide brief introductions of how super-resolution ultrasound works, how does it compare with other imaging modalities, and what are the tradeoffs and limitations for an audience who is not familiar with the technology.

Ultrasound Localization Microscopy Using Deep Neural Network

Noninvasive imaging of microvascular structures in deep tissues provides morphological and functional information for clinical diagnosis and monitoring. Ultrasound localization microscopy (ULM) is an emerging imaging technique that can generate microvascular structures with subwavelength diffraction resolution. However, the clinical utility of ULM is hindered by technical limitations, such as long data acquisition time, high microbubble (MB) concentration, and inaccurate localization. In this article, we propose a Swin transformer-based neural network to perform end-to-end mapping to implement MB localization. The performance of the proposed method was validated using synthetic and in vivo data using different quantitative metrics. The results indicate that our proposed network can achieve higher precision and better imaging capability than previously used methods. Furthermore, the computational cost of processing per frame is 3-4 times faster than traditional methods, which makes the real-time application of this technique feasible in the future.

Deep learning algorithm for predicting subacromial motion trajectory: Dynamic shoulder ultrasound analysis

Subacromial motion metrics can be extracted from dynamic shoulder ultrasonography, which is useful for identifying abnormal motion patterns in painful shoulders. However, frame-by-frame manual labeling of anatomical landmarks in ultrasound images is time consuming. The present study aims to investigate the feasibility of a deep learning algorithm for extracting subacromial motion metrics from dynamic ultrasonography. Dynamic ultrasound imaging was retrieved by asking 17 participants to perform cyclic shoulder abduction and adduction along the scapular plane, whereby the trajectory of the humeral greater tubercle (in relation to the lateral acromion) was depicted by the deep learning algorithm. Extraction of the subacromial motion metrics was conducted using a convolutional neural network (CNN) or a self-transfer learning-based (STL)-CNN with or without an autoencoder (AE). The mean absolute error (MAE) compared with the manually-labeled data (ground truth) served as the main outcome variable. Using eight-fold cross-validation, the average MAE was proven to be significantly higher in the group using CNN than in those using STL-CNN or STL-CNN+AE for the relative difference between the greater tubercle and lateral acromion on the horizontal axis. The MAE for the localization of the two aforementioned landmarks on the vertical axis also seemed to be enlarged in those using CNN compared with those using STL-CNN. In the testing dataset, the errors in relation to the ground truth for the minimal vertical acromiohumeral distance were 0.081-0.333 cm using CNN, compared with 0.002-0.007 cm using STL-CNN. We successfully demonstrated the feasibility of a deep learning algorithm for automatic detection of the greater tubercle and lateral acromion during dynamic shoulder ultrasonography. Our framework also demonstrated the capability of capturing the minimal vertical acromiohumeral distance, which is the most important indicator of subacromial motion metrics in daily clinical practice.

In vivo ocular microvasculature imaging in rabbits with 3D ultrasound localization microscopy

Morphological and hemodynamic changes in the ocular vasculature are important signs of various ocular diseases. The evaluation of the ocular microvasculature with high resolution is valuable in comprehensive diagnoses. However, it is difficult for current optical imaging techniques to visualize the posterior segment and retrobulbar microvasculature due to the limited penetration depth of light, particularly when the refractive medium is opaque. Thus, we have developed a 3D ultrasound localization microscopy (ULM) imaging method to visualize the ocular microvasculature in rabbits with micron-scale resolution. We used a 32 × 32 matrix array transducer (center frequency: 8 MHz) with a compounding plane wave sequence and microbubbles. Block-wise singular value decomposition spatiotemporal clutter filtering and block-matching 3D denoising were implemented to extract the flowing microbubble signals at different imaging depths with high signal-to-noise ratios. The center points of microbubbles were localized and tracked in 3D space to achieve the micro-angiography. The in vivo results demonstrate the ability of 3D ULM to visualize the microvasculature of the eye in rabbits, where vessels down to 54 μm were successfully revealed. Moreover, the microvascular maps indicated the morphological abnormalities in the eye with retinal detachment. This efficient modality shows potential for use in the diagnosis of ocular diseases.

A Data-Efficient Deep Learning Strategy for Tissue Characterization via Quantitative Ultrasound: Zone Training

Deep learning (DL) powered biomedical ultrasound imaging is an emerging research field where researchers adapt the image analysis capabilities of DL algorithms to biomedical ultrasound imaging settings. A major roadblock to wider adoption of DL powered biomedical ultrasound imaging is that acquisition of large and diverse datasets is expensive in clinical settings, which is a requirement for successful DL implementation. Hence, there is a constant need for developing data-efficient DL techniques to turn DL powered biomedical ultrasound imaging into reality. In this work, we develop a data-efficient DL training strategy for classifying tissues based on the ultrasonic backscattered RF data, i.e., quantitative ultrasound (QUS), which we named zone training. In zone training, we propose to divide the complete field of view of an ultrasound image into multiple zones associated with different regions of a diffraction pattern and then, train separate DL networks for each zone. The main advantage of zone training is that it requires less training data to achieve high accuracy. In this work, three different tissue-mimicking phantoms were classified by a DL network. The results demonstrated that zone training can require a factor of 2-3 less training data in low data regime to achieve similar classification accuracies compared to a conventional training strategy.

Laser diode photoacoustic point source detection: machine learning-based denoising and reconstruction

A new development in photoacoustic (PA) imaging has been the use of compact, portable and low-cost laser diodes (LDs), but LD-based PA imaging suffers from low signal intensity recorded by the conventional transducers. A common method to improve signal strength is temporal averaging, which reduces frame rate and increases laser exposure to patients. To tackle this problem, we propose a deep learning method that will denoise point source PA radio-frequency (RF) data before beamforming with a very few frames, even one. We also present a deep learning method to automatically reconstruct point sources from noisy pre-beamformed data. Finally, we employ a strategy of combined denoising and reconstruction, which can supplement the reconstruction algorithm for very low signal-to-noise ratio inputs.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Ultrasound Signal Processing: From Models to Deep Learning

Medical ultrasound imaging relies heavily on high-quality signal processing to provide reliable and interpretable image reconstructions. Conventionally, reconstruction algorithms have been derived from physical principles. These algorithms rely on assumptions and approximations of the underlying measurement model, limiting image quality in settings where these assumptions break down. Conversely, more sophisticated solutions based on statistical modeling or careful parameter tuning or derived from increased model complexity can be sensitive to different environments. Recently, deep learning-based methods, which are optimized in a data-driven fashion, have gained popularity. These model-agnostic techniques often rely on generic model structures and require vast training data to converge to a robust solution. A relatively new paradigm combines the power of the two: leveraging data-driven deep learning and exploiting domain knowledge. These model-based solutions yield high robustness and require fewer parameters and training data than conventional neural networks. In this work we provide an overview of these techniques from the recent literature and discuss a wide variety of ultrasound applications. We aim to inspire the reader to perform further research in this area and to address the opportunities within the field of ultrasound signal processing. We conclude with a future perspective on model-based deep learning techniques for medical ultrasound.

Ultrafast Ultrasound Localization Microscopy by Conditional Generative Adversarial Network

Ultrasound localization microscopy (ULM) overcomes the acoustic diffraction limit and enables the visualization of microvasculature at subwavelength resolution. However, challenges remain in ultrafast ULM implementation, where short data acquisition time, efficient data processing speed, and high imaging resolution need to be considered simultaneously. Recently, deep learning (DL)-based methods have exhibited potential in speeding up ULM imaging. Nevertheless, a certain number of ultrasound (US) data ( L frames) are still required to accumulate enough localized microbubble (MB) events, leading to an acquisition time within a time span of tens of seconds. To further speed up ULM imaging, in this article, we present a new DL-based method, termed as ULM-GAN. By using a modified conditional generative adversarial network (cGAN) framework, ULM-GAN is able to reconstruct a superresolution image directly from a temporal mean low-resolution (LR) image generated by averaging l -frame raw US images with l being significantly smaller than L . To evaluate the performance of ULM-GAN, a series of numerical simulations and phantom experiments are both implemented. The results of the numerical simulations demonstrate that when performing ULM imaging, ULM-GAN allows  ∼ 40 -fold reduction in data acquisition time and  ∼ 61 -fold reduction in computational time compared with the conventional Gaussian fitting method, without compromising spatial resolution according to the resolution scaled error (RSE). For the phantom experiments, ULM-GAN offers an implementation of ULM with ultrafast data acquisition time (  ∼ 0.33 s) and ultrafast data processing speed (  ∼ 0.60 s) that makes it promising to observe rapid biological activities in vivo.

Three-Dimensional Ultrasound Localization Microscopy with Bipartite Graph-Based Microbubble Pairing and Kalman-Filtering-Based Tracking on a 256-Channel Verasonics Ultrasound System with a 32 × 32 Matrix Array

Three-dimensional (3D) ultrasound localization microscopy (ULM) using a 2-D matrix probe and microbubbles (MBs) has been recently proposed to visualize microvasculature beyond the ultrasound diffraction limit in three spatial dimensions. However, 3D ULM suffers from several limitations: (1) high system complexity due to numerous channel counts, (2) complex MB flow dynamics in 3D, and (3) extremely long acquisition time. To reduce the system complexity while maintaining high image quality, we used a sub-aperture process to reduce received channel counts. To address the second issue, a 3D bipartite graph-based method with Kalman filtering-based tracking was used in this study for MB tracking. An MB separation approach was incorporated to separate high concentration MB data into multiple, sparser MB datasets, allowing better MB localization and tracking for a limited acquisition time. The proposed method was first validated in a flow channel phantom, showing improved spatial resolutions compared with the contrasted enhanced power Doppler image. Then the proposed method was evaluated with an in vivo chicken embryo brain dataset. Results showed that the reconstructed 3D super-resolution image achieved a spatial resolution of around 52 μm (smaller than the wavelength of around 200 μm). Microvessels that cannot be resolved clearly using localization only, can be well identified with the tailored 3D pairing and tracking algorithms. To sum up, the feasibility of the 3D ULM is shown, indicating the great possibility in clinical applications.

Curvelet Transform-Based Sparsity Promoting Algorithm for Fast Ultrasound Localization Microscopy

Ultrasound localization microscopy (ULM) based on microbubble (MB) localization was recently introduced to overcome the resolution limit of conventional ultrasound. However, ULM is currently challenged by the requirement for long data acquisition times to accumulate adequate MB events to fully reconstruct vasculature. In this study, we present a curvelet transform-based sparsity promoting (CTSP) algorithm that improves ULM imaging speed by recovering missing MB localization signal from data with very short acquisition times. CTSP was first validated in a simulated microvessel model, followed by the chicken embryo chorioallantoic membrane (CAM), and finally, in the mouse brain. In the simulated microvessel study, CTSP robustly recovered the vessel model to achieve an 86.94% vessel filling percentage from a corrupted image with only 4.78% of the true vessel pixels. In the chicken embryo CAM study, CTSP effectively recovered the missing MB signal within the vasculature, leading to marked improvement in ULM imaging quality with a very short data acquisition. Taking the optical image as reference, the vessel filling percentage increased from 2.7% to 42.2% using 50ms of data acquisition after applying CTSP. CTSP used 80% less time to achieve the same 90% maximum saturation level as compared with conventional MB localization. We also applied CTSP on the microvessel flow speed maps and found that CTSP was able to use only 1.6s of microbubble data to recover flow speed images that have similar qualities as those constructed using 33.6s of data. In the mouse brain study, CTSP was able to reconstruct the majority of the cerebral vasculature using 1-2s of data acquisition. Additionally, CTSP only needed 3.2s of microbubble data to generate flow velocity maps that are comparable to those using 129.6s of data. These results suggest that CTSP can facilitate fast and robust ULM imaging especially under the circumstances of inadequate microbubble localizations.

Super-Resolution Ultrasound Through Sparsity-Based Deconvolution and Multi-Feature Tracking

Ultrasound super-resolution imaging through localisation and tracking of microbubbles can achieve sub-wave-diffraction resolution in mapping both micro-vascular structure and flow dynamics in deep tissue in vivo. Currently, it is still challenging to achieve high accuracy in localisation and tracking particularly with limited imaging frame rates and in the presence of high bubble concentrations. This study introduces microbubble image features into a Kalman tracking framework, and makes the framework compatible with sparsity-based deconvolution to address these key challenges. The performance of the method is evaluated on both simulations using individual bubble signals segmented from in vivo data and experiments on a mouse brain and a human lymph node. The simulation results show that the deconvolution not only significantly improves the accuracy of isolating overlapping bubbles, but also preserves some image features of the bubbles. The combination of such features with Kalman motion model can achieve a significant improvement in tracking precision at a low frame rate over that using the distance measure, while the improvement is not significant at the highest frame rate. The in vivo results show that the proposed framework generates SR images that are significantly different from the current methods with visual improvement, and is more robust to high bubble concentrations and low frame rates.

Deep learning for dose assessment in radiotherapy by the super-localization of vaporized nanodroplets in high frame rate ultrasound imaging

Objective. External beam radiotherapy is aimed to precisely deliver a high radiation dose to malignancies, while optimally sparing surrounding healthy tissues. With the advent of increasingly complex treatment plans, the delivery should preferably be verified by quality assurance methods. Recently, online ultrasound imaging of vaporized radiosensitive nanodroplets was proposed as a promising tool for in vivo dosimetry in radiotherapy. Previously, the detection of sparse vaporization events was achieved by applying differential ultrasound (US) imaging followed by intensity thresholding using subjective parameter tuning, which is sensitive to image artifacts. Approach. A generalized deep learning solution (i.e. BubbleNet) is proposed to localize vaporized nanodroplets on differential US frames, while overcoming the aforementioned limitation. A 5-fold cross-validation was performed on a diversely composed 5747-frame training/validation dataset by manual segmentation. BubbleNet was then applied on a test dataset of 1536 differential US frames to evaluate dosimetric features. The intra-observer variability was determined by scoring the Dice similarity coefficient (DSC) on 150 frames segmented twice. Additionally, the BubbleNet generalization capability was tested on an external test dataset of 432 frames acquired by a phased array transducer at a much lower ultrasound frequency and reconstructed with unconventional pixel dimensions with respect to the training dataset. Main results. The median DSC in the 5-fold cross validation was equal to ∼0.88, which was in line with the intra-observer variability (=0.86). Next, BubbleNet was employed to detect vaporizations in differential US frames obtained during the irradiation of phantoms with a 154 MeV proton beam or a 6 MV photon beam. BubbleNet improved the bubble-count statistics by ∼30% compared to the earlier established intensity-weighted thresholding. The proton range was verified with a −0.8 mm accuracy. Significance. BubbleNet is a flexible tool to localize individual vaporized nanodroplets on experimentally acquired US images, which improves the sensitivity compared to former thresholding-weighted methods.

Deep Learning-Based Microbubble Localization for Ultrasound Localization Microscopy

Ultrasound localization microscopy (ULM) is an emerging vascular imaging technique that overcomes the resolution-penetration compromise of ultrasound imaging. Accurate and robust microbubble (MB) localization is essential for successful ULM. In this study, we present a deep learning (DL)-based localization technique that uses both Field-II simulation and in vivo chicken embryo chorioallantoic membrane (CAM) data for training. Both radio frequency (RF) and in-phase and quadrature (IQ) data were tested in this study. The simulation experiment shows that the proposed DL-based localization was able to reduce both missing MB localization rate and MB localization error. In general, RF data showed better performance than IQ. For the in vivo CAM study with high MB concentration, DL-based localization was able to reduce the vessel MB saturation time by more than 50% compared to conventional localization. In addition, we propose a DL-based framework for real-time visualization of the high-resolution microvasculature. The findings of this article support the use of DL for more robust and faster MB localization, especially under high MB concentrations. The results indicate that further improvement could be achieved by incorporating temporal information of the MB data.

Subwavelength ultrasonic imaging using a deep convolutional neural network trained on structural noise

Subwavelength ultrasonic imaging (SUI) can detect subwavelength flaws beyond the diffraction limit, however, SUI sometimes fails to clearly reveal flaws in C-scans when the signal-to-noise ratio (SNR) is low. In this work, a convolutional neural network (CNN) that takes structural noise into account is developed for SUI to distinguish flaw echoes from structural noise. The network contains a regression CNN for learning features from the structural noise and a learnable soft thresholding layer for classification. Experiments show that the proposed method performs well for imaging subwavelength flaws at different depths and of different sizes. It achieved an F₁ score of 97.69 ± 1.56% in detecting flaws as compared to the enhanced ultrasonic flaw detection method with time-dependent threshold. As an example of general application of the method, we also performed SUI on natural flaws in a spheroidal graphite cast iron specimen. The results show that the method can achieve SUI without a theoretical backscattering model and is not limited by noise distribution, multiple scattering, or complex microstructures. Furthermore, the network does not need to prepare flaw echoes for training.

ApodNet: Learning for High Frame Rate Synthetic Transmit Aperture Ultrasound Imaging

Two-way dynamic focusing in synthetic transmit aperture (STA) beamforming can benefit high-quality ultrasound imaging with higher lateral spatial resolution and contrast resolution. However, STA requires the complete dataset for beamforming in a relatively low frame rate and transmit power. This paper proposes a deep-learning architecture to achieve high frame rate STA imaging with two-way dynamic focusing. The network consists of an encoder and a joint decoder. The encoder trains a set of binary weights as the apodizations of the high-frame-rate plane wave transmissions. In this respect, we term our network ApodNet. The decoder can recover the complete dataset from the acquired channel data to achieve dynamic transmit focusing. We evaluate the proposed method by simulations at different levels of noise and in-vivo experiments on the human biceps brachii and common carotid artery. The experimental results demonstrate that ApodNet provides a promising strategy for high frame rate STA imaging, obtaining comparable lateral resolution and contrast resolution with four-times higher frame rate than conventional STA imaging in the in-vivo experiments. Particularly, ApodNet improves contrast resolution of the hypoechoic targets with much shorter computational time when compared with other high-frame-rate methods in both simulations and in-vivo experiments.

Compressed Sensing-Based Super-Resolution Ultrasound Imaging for Faster Acquisition and High Quality Images

GOAL

Typical SRUS images are reconstructed by localizing ultrasound microbubbles (MBs) injected in a vessel using normalized 2-dimensional cross-correlation (2DCC) between MBs signals and the point spread function of the system. However, current techniques require isolated MBs in a confined area due to inaccurate localization of densely populated MBs. To overcome this limitation, we developed the ℓ1-homotopy based compressed sensing (L1H-CS) based SRUS imaging technique which localizes densely populated MBs to visualize microvasculature in vivo.

METHODS

To evaluate the performance of L1H-CS, we compared the performance of 2DCC, interior-point method based compressed sensing (CVX-CS), and L1H-CS algorithms. Localization efficiency was compared using axially and laterally aligned point targets (PTs) with known distances and randomly distributed PTs generated by simulation. We developed post-processing techniques including clutter reduction, noise equalization, motion compensation, and spatiotemporal noise filtering for in vivo imaging. We then validated the capabilities of L1H-CS based SRUS imaging technique with high-density MBs in a mouse tumor model, kidney, and zebrafish dorsal trunk, and brain.

RESULTS

Compared to 2DCC and CVX-CS algorithms, L1H-CS achieved faster data acquisition time and considerable improvement in SRUS image quality.

CONCLUSIONS AND SIGNIFICANCE

These results demonstrate that the L1H-CS based SRUS imaging technique has the potential to examine microvasculature with reduced acquisition and reconstruction time to acquire enhanced SRUS image quality, which may be necessary to translate it into clinics.

Ultrasound microvasculature imaging with entropy-based radiality super-resolution (ERSR)

Objective:Microvasculature is highly relevant to the occurrence and development of pathologies such as cancer and diabetes. Ultrasound localization microscopy (ULM) has bypassed the diffraction limit and demonstrated its great potential to provide new imaging modality and establish new diagnostic criteria in clinical application. However, sparse microbubble distribution can be a significant bottleneck for improving temporal resolution, even for further clinical translation. Other important challenges for ULM to tackle in clinic also include high microbubble concentration and low frame rate.Approach:As part of the efforts to facilitate clinical translation, this paper focused on the low frame rate and the high microbubble distribution issue and proposed a new super-resolution imaging strategy called entropy-based radiality super-resolution (ERSR). The feasibility of ERSR is validated with simulations, phantom experiment and contrast-enhanced ultrasound scan of rabbit sciatic nerve with clinical accessible ultrasound system.Main results:ERSR can achieve 10 times improvement in spatial resolution compared to conventional ultrasound imaging, higher temporal resolution (∼10 times higher) and contrast-to-noise ratio under high-density microbubbles, compared with ULM under low-density microbubbles.Significance:We conclude ERSR could be a valuable imaging tool with high spatio-temporal resolution for clinical diagnosis and assessment of diseases potentially.

Faster super-resolution ultrasound imaging with a deep learning model for tissue decluttering and contrast agent localization

Super-resolution ultrasound (SR-US) imaging allows visualization of microvascular structures as small as tens of micrometers in diameter. However, use in the clinical setting has been impeded in part by ultrasound (US) acquisition times exceeding a breath-hold and by the need for extensive offline computation. Deep learning techniques have been shown to be effective in modeling the two more computationally intensive steps of microbubble (MB) contrast agent detection and localization. Performance gains by deep networks over conventional methods are more than two orders of magnitude and in addition the networks can localize overlapping MBs. The ability to separate overlapping MBs allows use of higher contrast agent concentrations and reduces US image acquisition time. Herein we propose a fully convolutional neural network (CNN) architecture to perform the operations of MB detection as well as localization in a single model. Termed SRUSnet, the network is based on the MobileNetV3 architecture modified for 3-D input data, minimal convergence time, and high-resolution data output using a flexible regression head. Also, we propose to combine linear B-mode US imaging and nonlinear contrast pulse sequencing (CPS) which has been shown to increase MB detection and further reduce the US image acquisition time. The network was trained within silicodata and tested onin vitrodata from a tissue-mimicking flow phantom, and onin vivodata from the rat hind limb (N = 3). Images were collected with a programmable US system (Vantage 256, Verasonics Inc., Kirkland, WA) using an L11-4v linear array transducer. The network exceeded 99.9% detection accuracy onin silicodata. The average localization accuracy was smaller than the resolution of a pixel (i.e.λ/8). The average processing time on a Nvidia GeForce 2080Ti GPU was 64.5 ms for a 128 × 128-pixel image.

In Vivo Motion Correction in Super-Resolution Imaging of Rat Kidneys

Super-resolution (SR) imaging has the potential of visualizing the microvasculature down to the 10- [Formula: see text] level, but motion induced by breathing, heartbeats, and muscle contractions are often significantly above this level. This article, therefore, introduces a method for estimating tissue motion and compensating for this. The processing pipeline is described and validated using Field II simulations of an artificial kidney. In vivo measurements were conducted using a modified bk5000 research scanner (BK Medical, Herlev, Denmark) with a BK 9009 linear array probe employing a pulse amplitude modulation scheme. The left kidney of ten Sprague-Dawley rats was scanned during open laparotomy. A 1:10 diluted SonoVue contrast agent (Bracco, Milan, Italy) was injected through a jugular vein catheter at 100 [Formula: see text]/min. Motion was estimated using speckle tracking and decomposed into contributions from the heartbeats, breathing, and residual motion. The estimated peak motions and their precisions were: heart: axial- [Formula: see text] and lateral- [Formula: see text], breathing: axial- [Formula: see text] and lateral- [Formula: see text], and residual: axial-30 [Formula: see text] and lateral-90 [Formula: see text]. The motion corrected microbubble tracks yielded SR images of both bubble density and blood vector velocity. The estimation was, thus, sufficiently precise to correct shifts down to the 10- [Formula: see text] capillary level. Similar results were found in the other kidney measurements with a restoration of resolution for the small vessels demonstrating that motion correction in 2-D can enhance SR imaging quality.

A Deep Learning Framework for Spatiotemporal Ultrasound Localization Microscopy

Ultrasound Localization Microscopy (ULM) can resolve the microvascular bed down to a few micrometers. To achieve such performance, microbubble contrast agents must perfuse the entire microvascular network. Microbubbles are then located individually and tracked over time to sample individual vessels, typically over hundreds of thousands of images. To overcome the fundamental limit of diffraction and achieve a dense reconstruction of the network, low microbubble concentrations must be used, which leads to acquisitions lasting several minutes. Conventional processing pipelines are currently unable to deal with interference from multiple nearby microbubbles, further reducing achievable concentrations. This work overcomes this problem by proposing a Deep Learning approach to recover dense vascular networks from ultrasound acquisitions with high microbubble concentrations. A realistic mouse brain microvascular network, segmented from 2-photon microscopy, was used to train a three-dimensional convolutional neural network (CNN) based on a V-net architecture. Ultrasound data sets from multiple microbubbles flowing through the microvascular network were simulated and used as ground truth to train the 3D CNN to track microbubbles. The 3D-CNN approach was validated in silico using a subset of the data and in vivo in a rat brain. In silico, the CNN reconstructed vascular networks with higher precision (81%) than a conventional ULM framework (70%). In vivo, the CNN could resolve micro vessels as small as 10 μ m with an improvement in resolution when compared against a conventional approach.

Super-resolution ultrasound localization microscopy based on a high frame-rate clinical ultrasound scanner: an in-human feasibility study

Non-invasive detection of microvascular alterations in deep tissuesin vivoprovides critical information for clinical diagnosis and evaluation of a broad-spectrum of pathologies. Recently, the emergence of super-resolution ultrasound localization microscopy (ULM) offers new possibilities for clinical imaging of microvasculature at capillary level. Currently, the clinical utility of ULM on clinical ultrasound scanners is hindered by the technical limitations, such as long data acquisition time, high microbubble (MB) concentration, and compromised tracking performance associated with low imaging frame-rate. Here we present a robust in-human ULM on a high frame-rate (HFR) clinical ultrasound scanner to achieve super-resolution microvessel imaging using a short acquisition time (<10 s). Ultrasound MB data were acquired from different human tissues, including a healthy liver and a diseased liver with acute-on-chronic liver failure, a kidney, a pancreatic tumor, and a breast mass using an HFR clinical scanner. By leveraging the HFR and advanced processing techniques including sub-pixel motion registration, MB signal separation, and Kalman filter-based tracking, MBs can be robustly localized and tracked for ULM under the circumstances of relatively high MB concentration associated with standard clinical MB administration and limited data acquisition time in humans. Subtle morphological and hemodynamic information in microvasculature were shown based on data acquired with single breath-hold and free-hand scanning. Compared with contrast-enhanced power Doppler generated based on the same MB dataset, ULM showed a 5.7-fold resolution improvement in a vessel based on a linear transducer, and provided a wide-range blood flow speed measurement that is Doppler angle-independent. Microvasculatures with complex hemodynamics can be well-differentiated at super-resolution in both normal and pathological tissues. This preliminary study implemented the ultrafast in-human ULM in various human tissues based on a clinical scanner that supports HFR imaging, indicating the potentials of the technique for various clinical applications. However, rigorous validation of the technique in imaging human microvasculature (especially for those tiny vessel structure), preferably with a gold standard, is still required.

Current Development and Applications of Super-Resolution Ultrasound Imaging

Abnormal changes of the microvasculature are reported to be key evidence of the development of several critical diseases, including cancer, progressive kidney disease, and atherosclerotic plaque. Super-resolution ultrasound imaging is an emerging technology that can identify the microvasculature noninvasively, with unprecedented spatial resolution beyond the acoustic diffraction limit. Therefore, it is a promising approach for diagnosing and monitoring the development of diseases. In this review, we introduce current super-resolution ultrasound imaging approaches and their preclinical applications on different animals and disease models. Future directions and challenges to overcome for clinical translations are also discussed.

Fast super-resolution ultrasound microvessel imaging using spatiotemporal data with deep fully convolutional neural network

Ultrasound localization microscopy (ULM) has been proposed to image microvasculature beyond the ultrasound diffraction limit. Although ULM can attain microvascular images with a sub-diffraction resolution, long data acquisition time and processing time are the critical limitations. Deep learning-based ULM (deep-ULM) has been proposed to mitigate these limitations. However, microbubble (MB) localization used in deep-ULMs is currently based on spatial information without the use of temporal information. The highly spatiotemporally coherent MB signals provide a strong feature that can be used to differentiate MB signals from background artifacts. In this study, a deep neural network was employed and trained with spatiotemporal ultrasound datasets to better identify the MB signals by leveraging both the spatial and temporal information of the MB signals. Training, validation and testing datasets were acquired from MB suspension to mimic the realistic intensity-varying and moving MB signals. The performance of the proposed network was first demonstrated in the chicken embryo chorioallantoic membrane dataset with an optical microscopic image as the reference standard. Substantial improvement in spatial resolution was shown for the reconstructed super-resolved images compared with power Doppler images. The full-width-half-maximum (FWHM) of a microvessel was improved from 133μm to 35μm, which is smaller than the ultrasound wavelength (73μm). The proposed method was further tested in anin vivohuman liver data. Results showed the reconstructed super-resolved images could resolve a microvessel of nearly 170μm (FWHM). Adjacent microvessels with a distance of 670μm, which cannot be resolved with power Doppler imaging, can be well-separated with the proposed method. Improved contrast ratios using the proposed method were shown compared with that of the conventional deep-ULM method. Additionally, the processing time to reconstruct a high-resolution ultrasound frame with an image size of 1024 × 512 pixels was around 16 ms, comparable to state-of-the-art deep-ULMs.