High Spatial-Temporal Resolution Reconstruction of Plane-Wave Ultrasound Images With a Multichannel Multiscale Convolutional Neural Network

Improved Limited-View Ultrasound Tomography via Machine Learning

Tomographic reconstruction is used extensively in medicine, nondestructive testing (NDT), and geology. In an ideal situation, where measurements are taken at all angles around an object, known as full-view configuration, a full reconstruction of the object can be produced. One of the major issues faced in tomographic imaging is when measurements cannot be taken freely around the object under inspection. This may be caused by the size and geometry of the object or difficulty accessing from particular directions. The resulting limited view (LV) transducer configuration leads to a large deterioration in image quality; thus, it is very beneficial to employ a compensation algorithm. At present, the most effective compensation algorithms require a large amount of computing power or a bespoke case-by-case approach, often with numerous arbitrary constants which must be tuned for a specific application. This work proposes a machine learning (ML)-based approach to perform the LV compensation. The model is based around an autoencoder (AE) architecture. It is trained on an artificial dataset, taking advantage of the ability to generate arbitrary LV images given a full view input. The approach is evaluated on ten laser-scanned corrosion maps and the results compared to positivity regularization-a LV compensation algorithm similar in the speed of execution and generalization potential. The algorithms are compared for root mean squared error (RMSE) across the image and maximum absolute error (MAE). Furthermore, they are visually compared for subjective quality. Compared to the conventional algorithm, the ML-based approach improves on the MAE in eight out of the ten cases. The conventional approach performs better on mean RMSE, which is explained by the ML outputting an inaccurate background level, which is not a critical ability. Most importantly, the visual inspection of outputs shows the ML approach reconstructs the images better, especially in the case of irregular corrosion patches. Compared to LV images, the ML method improves both the RMSE and MAE by 41%.

Virtual multiplexed immunofluorescence staining from non-antibody-stained fluorescence imaging for gastric cancer prognosis

BACKGROUND

Multiplexed immunofluorescence (mIF) staining, such as CODEX and MIBI, holds significant clinical value for various fields, such as disease diagnosis, biological research, and drug development. However, these techniques are often hindered by high time and cost requirements.

METHODS

Here we present a Multimodal-Attention-based virtual mIF Staining (MAS) system that utilises a deep learning model to extract potential antibody-related features from dual-modal non-antibody-stained fluorescence imaging, specifically autofluorescence (AF) and DAPI imaging. The MAS system simultaneously generates predictions of mIF with multiple survival-associated biomarkers in gastric cancer using self- and multi-attention learning mechanisms.

FINDINGS

Experimental results with 180 pathological slides from 94 patients with gastric cancer demonstrate the efficiency and consistent performance of the MAS system in both cancer and noncancer gastric tissues. Furthermore, we showcase the prognostic accuracy of the virtual mIF images of seven gastric cancer related biomarkers, including CD3, CD20, FOXP3, PD1, CD8, CD163, and PD-L1, which is comparable to those obtained from the standard mIF staining.

INTERPRETATION

The MAS system rapidly generates reliable multiplexed staining, greatly reducing the cost of mIF and improving clinical workflow.

FUNDING

Stanford 2022 HAI Seed Grant; National Institutes of Health 1R01CA256890.

Improvement in Multi-Angle Plane Wave Image Quality Using Minimum Variance Beamforming with Adaptive Signal Coherence

For ultrasound multi-angle plane wave (PW) imaging, the coherent PW compounding (CPWC) method provides limited image quality because of its conventional delay-and-sum beamforming. The delay-multiply-and-sum (DMAS) method is a coherence-based algorithm that improves image quality by introducing signal coherence among either receiving channels or PW transmit angles into the image output. The degree of signal coherence in DMAS is conventionally a global value for the entire image and thus the image resolution and contrast in the target region improves at the cost of speckle quality in the background region. In this study, the adaptive DMAS (ADMAS) is proposed such that the degree of signal coherence relies on the local characteristics of the image region to maintain the background speckle quality and the corresponding contrast-to-noise ratio (CNR). Subsequently, the ADMAS algorithm is further combined with minimum variance (MV) beamforming to increase the image resolution. The optimal MV estimation is determined to be in the direction of the PW transmit angle (Tx) for multi-angle PW imaging. Our results show that, using the PICMUS dataset, TxMV-ADMAS beamforming significantly improves the image quality compared with CPWC. When the p value is globally fixed to 2 as in conventional DMAS, though the main-lobe width and the image contrast in the experiments improve from 0.57 mm and 27.0 dB in CPWC, respectively, to 0.24 mm and 38.0 dB, the corresponding CNR decreases from 12.8 to 11.3 due to the degraded speckle quality. With the proposed ADMAS algorithm, however, the adaptive p value in DMAS beamforming helps to restore the CNR value to the same level of CPWC while the improvement in image resolution and contrast remains evident.

ARU-GAN: U-shaped GAN based on Attention and Residual connection for super-resolution reconstruction

Plane-wave ultrasound imaging technology offers high-speed imaging but lacks image quality. To improve the image spatial resolution, beam synthesis methods are used, which often compromise the temporal resolution. Herein, we propose ARU-GAN, a super-resolution reconstruction model based on residual connectivity and attention mechanisms, to address this issue. ARU-GAN comprises a Full-scale Skip-connection U-shaped Generator (FSUG) with an attention mechanism and a Residual Attention Patch Discriminator (RAPD). The former captures global and local features of the image by using full-scale skip-connections and attention mechanisms. The latter focuses on changes in the image at different scales to enhance its discriminative ability at the patch level. ARU-GAN was trained using a combined loss function on the Plane-Wave Imaging Challenge in Medical Ultrasound (PICMUS) 2016 dataset, which includes three types of targets: point targets, cyst targets, and in-vivo targets. Compared to Coherent Plane-Wave Compounding (CPWC), ARU-GAN achieved a reduction in Full Width at Half Maximum (FWHM) by 5.78%-20.30% on point targets, improved Contrast (CR) by 7.59-11.29 percentage points, and Contrast to Noise Ratio (CNR) by 30.58%-45.22% on cyst targets. On in-vivo target, ARU-GAN improved the Peak Signal-to-Noise Ratio (PSNR) by 11.94%, the Complex-Wavelet Structural Similarity Index Measurement (CW-SSIM) by 17.11%, and the Normalized Cross Correlation (NCC) by at least 2.17% compared to existing deep learning methods. In conclusion, ARU-GAN is a promising model for the super-resolution reconstruction of plane-wave medical ultrasound images. It provides a novel solution for improving image quality, which is essential for clinical practice.

A robust cascaded deep neural network for image reconstruction of single plane wave ultrasound RF data

Reconstruction of ultrasound data from single plane wave Radio Frequency (RF) data is a challenging task. The traditional Delay and Sum (DAS) method produces an image with low resolution and contrast, if employed with RF data from only a single plane wave. A Coherent Compounding (CC) method that reconstructs the image by coherently summing the individual DAS images was proposed to enhance the image quality. However, CC relies on a large number of plane waves to accurately sum the individual DAS images, hence it produces high quality images but with low frame rate that may not be suitable for time-demanding applications. Therefore, there is a need for a method that can create a high quality image with higher frame rates. Furthermore, the method needs to be robust against the input transmission angle of the plane wave. To reduce the method's dependence on the input angle, we propose to unify the RF data at different angles by learning a linear data transformation from different angled data to a common, 0° data. We further propose a cascade of two independent neural networks to reconstruct an image, similar in quality to CC, by making use of a single plane wave. The first network, denoted as "PixelNet", is a fully Convolutional Neural Network (CNN) which takes in the transformed time-delayed RF data as input. PixelNet learns optimal pixel weights that get element-wise multiplied with the single angle DAS image. The second network is a conditional Generative Adversarial Network (cGAN) which is used to further enhance the image quality. Our networks were trained on the publicly available PICMUS and CPWC datasets and evaluated on a completely separate, CUBDL dataset obtained from different acquisition settings than the training dataset. The results thus obtained on the testing dataset, demonstrate the networks' ability to generalize well on unseen data, with frame rates better than the CC method. This paves the way for applications that require high-quality images reconstructed at higher frame rates.

Extended aperture image reconstruction for plane-wave imaging

B-mode images undergo degradation in the boundary region because of the limited number of elements in the ultrasound probe. Herein, a deep learning-based extended aperture image reconstruction method is proposed to reconstruct a B-mode image with an enhanced boundary region. The proposed network can reconstruct an image using pre-beamformed raw data received from the half-aperture of the probe. To generate a high-quality training target without degradation in the boundary region, the target data were acquired using the full-aperture. Training data were acquired from an experimental study using a tissue-mimicking phantom, vascular phantom, and simulation of random point scatterers. Compared with plane-wave images from delay and sum beamforming, the proposed extended aperture image reconstruction method achieves improvement at the boundary region in terms of the multi-scale structure of similarity and peak signal-to-noise ratio by 8% and 4.10 dB in resolution evaluation phantom, 7% and 3.15 dB in contrast speckle phantom, and 5% and 3 dB in in vivo study of carotid artery imaging. The findings in this study prove the feasibility of a deep learning-based extended aperture image reconstruction method for boundary region improvement.

Ultrasonic imaging using conditional generative adversarial networks

The Full Matrix Capture (FMC) and Total Focusing Method (TFM) combination is often considered the gold standard in ultrasonic nondestructive testing, however it may be impractical due to the amount of time required to gather and process the FMC, particularly for high cadence inspections. This study proposes replacing conventional FMC acquisition and TFM processing with a single zero-degree plane wave (PW) insonification and a conditional Generative Adversarial Network (cGAN) trained to produce TFM-like images. Three models with different cGAN architectures and loss formulations were tested in different scenarios. Their performances were compared with conventional TFM computed from FMC. The proposed cGANs were able to recreate TFM-like images with the same resolution while improving the contrast in more than 94% of the reconstructions in comparison with conventional TFM reconstructions. Indeed, thanks to the use of a bias in the cGANs' training, the contrast was systematically increased through a reduction of the background noise level and the elimination of some artifacts. Finally, the proposed method led to a reduction of the computation time and file size by a factor of 120 and 75, respectively.

Ultrasound Signal Processing: From Models to Deep Learning

Medical ultrasound imaging relies heavily on high-quality signal processing to provide reliable and interpretable image reconstructions. Conventionally, reconstruction algorithms have been derived from physical principles. These algorithms rely on assumptions and approximations of the underlying measurement model, limiting image quality in settings where these assumptions break down. Conversely, more sophisticated solutions based on statistical modeling or careful parameter tuning or derived from increased model complexity can be sensitive to different environments. Recently, deep learning-based methods, which are optimized in a data-driven fashion, have gained popularity. These model-agnostic techniques often rely on generic model structures and require vast training data to converge to a robust solution. A relatively new paradigm combines the power of the two: leveraging data-driven deep learning and exploiting domain knowledge. These model-based solutions yield high robustness and require fewer parameters and training data than conventional neural networks. In this work we provide an overview of these techniques from the recent literature and discuss a wide variety of ultrasound applications. We aim to inspire the reader to perform further research in this area and to address the opportunities within the field of ultrasound signal processing. We conclude with a future perspective on model-based deep learning techniques for medical ultrasound.

Improving Lateral Resolution in 3-D Imaging With Micro-beamforming Through Adaptive Beamforming by Deep Learning

There is an increased desire for miniature ultrasound probes with small apertures to provide volumetric images at high frame rates for in-body applications. Satisfying these increased requirements makes simultaneous achievement of a good lateral resolution a challenge. As micro-beamforming is often employed to reduce data rate and cable count to acceptable levels, receive processing methods that try to improve spatial resolution will have to compensate the introduced reduction in focusing. Existing beamformers do not realize sufficient improvement and/or have a computational cost that prohibits their use. Here we propose the use of adaptive beamforming by deep learning (ABLE) in combination with training targets generated by a large aperture array, which inherently has better lateral resolution. In addition, we modify ABLE to extend its receptive field across multiple voxels. We illustrate that this method improves lateral resolution both quantitatively and qualitatively, such that image quality is improved compared with that achieved by existing delay-and-sum, coherence factor, filtered-delay-multiplication-and-sum and Eigen-based minimum variance beamformers. We found that only in silica data are required to train the network, making the method easily implementable in practice.

Deep reconstruction of high-quality ultrasound images from raw plane-wave data: A simulation and in vivo study

This paper presents a novel beamforming approach based on deep learning to get closer to the ideal Point Spread Function (PSF) in Plane-Wave Imaging (PWI). The proposed approach is designed to reconstruct a high-quality version of Tissue Reflectivity Function (TRF) from echo traces acquired by transducer elements using only a single plane-wave transmission. In this approach, first, a model for the TRF is introduced by setting the imaging PSF as an isotropic (i.e., circularly symmetric) 2D Gaussian kernel convolved with a cosine function. Then, a mapping function between the pre-beamformed Radio-Frequency (RF) channel data and the proposed output is constructed using deep learning. Network architecture contains multi-resolution decomposition and reconstruction using wavelet transform for effective recovery of high-frequency content of the desired output. We exploit step by step training from coarse (mean square error) to fine (ℓ_0.2) loss functions. The proposed method is trained on 1174 simulation ultrasound data with the ground-truth echogenicity map extracted from real photographic images. The performance of the trained network is evaluated on the publicly available simulation and in vivo test data without any further fine-tuning. Simulation test results show an improvement of 37.5% and 65.8% in terms of axial and lateral resolution as compared to Delay-And-Sum (DAS) results, respectively. The contrast is also improved by 33.7% in comparison to DAS. Furthermore, the reconstructed in vivo images confirm that the trained mapping function does not need any fine-tuning in the new domain. Therefore, the proposed approach maintains high resolution, contrast, and framerate simultaneously.

Reconstruction for plane-wave ultrasound imaging using modified U-Net-based beamformer

An image reconstruction method that can simultaneously provide high image quality and frame rate is necessary for diagnosis on cardiovascular imaging but is challenging for plane-wave ultrasound imaging. To overcome this challenge, an end-to-end ultrasound image reconstruction method is proposed for reconstructing a high-resolution B-mode image from radio frequency (RF) data. A modified U-Net architecture that adopts EfficientNet-B5 and U-Net as the encoder and decoder parts, respectively, is proposed as a deep learning beamformer. The training data comprise pairs of pre-beamformed RF data generated from random scatterers with random amplitudes and corresponding high-resolution target data generated from coherent plane-wave compounding (CPWC). To evaluate the performance of the proposed beamforming model, simulation and experimental data are used for various beamformers, such as delay-and-sum (DAS), CPWC, and other deep learning beamformers, including U-Net and EfficientNet-B0. Compared with single plane-wave imaging with DAS, the proposed beamforming model reduces the lateral full width at half maximum by 35% for simulation and 29.6% for experimental data and improves the contrast-to-noise ratio and peak signal-to-noise ratio, respectively, by 6.3 and 9.97 dB for simulation, 2.38 and 3.01 dB for experimental data, and 3.18 and 1.03 dB for in vivo data. Furthermore, the computational complexity of the proposed beamforming model is four times less than that of the U-Net beamformer. The study results demonstrate that the proposed ultrasound image reconstruction method employing a deep learning beamformer, trained by the RF data from scatterers, can reconstruct a high-resolution image with a high frame rate for single plane-wave ultrasound imaging.

CNN-Based Image Reconstruction Method for Ultrafast Ultrasound Imaging

Ultrafast ultrasound (US) revolutionized biomedical imaging with its capability of acquiring full-view frames at over 1 kHz, unlocking breakthrough modalities such as shear-wave elastography and functional US neuroimaging. Yet, it suffers from strong diffraction artifacts, mainly caused by grating lobes, sidelobes, or edge waves. Multiple acquisitions are typically required to obtain a sufficient image quality, at the cost of a reduced frame rate. To answer the increasing demand for high-quality imaging from single unfocused acquisitions, we propose a two-step convolutional neural network (CNN)-based image reconstruction method, compatible with real-time imaging. A low-quality estimate is obtained by means of a backprojection-based operation, akin to conventional delay-and-sum beamforming, from which a high-quality image is restored using a residual CNN with multiscale and multichannel filtering properties, trained specifically to remove the diffraction artifacts inherent to ultrafast US imaging. To account for both the high dynamic range and the oscillating properties of radio frequency US images, we introduce the mean signed logarithmic absolute error (MSLAE) as a training loss function. Experiments were conducted with a linear transducer array, in single plane-wave (PW) imaging. Trainings were performed on a simulated dataset, crafted to contain a wide diversity of structures and echogenicities. Extensive numerical evaluations demonstrate that the proposed approach can reconstruct images from single PWs with a quality similar to that of gold-standard synthetic aperture imaging, on a dynamic range in excess of 60 dB. In vitro and in vivo experiments show that trainings carried out on simulated data perform well in experimental settings.

Improving Image Quality for Single-Angle Plane Wave Ultrasound Imaging With Convolutional Neural Network Beamformer

Ultrafast ultrasound imaging based on plane wave (PW) compounding has been proposed for use in various clinical and preclinical applications, including shear wave imaging and super resolution blood flow imaging. Because the image quality afforded by PW imaging is highly dependent on the number of PW angles used for compounding, a tradeoff between image quality and frame rate occurs. In the present study, a convolutional neural network (CNN) beamformer based on a combination of the GoogLeNet and U-Net architectures was developed to replace the conventional delay-and-sum (DAS) algorithm to obtain high-quality images at a high frame rate. RF channel data are used as the inputs for the CNN beamformers. The outputs are in-phase and quadrature data. Simulations and phantom experiments revealed that the images predicted by the CNN beamformers had higher resolution and contrast than those predicted by conventional single-angle PW imaging with the DAS approach. In in vivo studies, the contrast-to-noise ratios (CNRs) of carotid artery images predicted by the CNN beamformers using three or five PWs as ground truths were approximately 12 dB in the transverse view, considerably higher than the CNR obtained using the DAS beamformer (3.9 dB). Most tissue speckle information was retained in the in vivo images produced by the CNN beamformers. In conclusion, only a single PW at 0° was fired, but the quality of the output image was proximal to that of an image generated using three or five PW angles. In other words, the quality-frame rate tradeoff of coherence compounding could be mitigated through the use of the proposed CNN for beamforming.

Training Deep Network Ultrasound Beamformers With Unlabeled In Vivo Data

Conventional delay-and-sum (DAS) beamforming is highly efficient but also suffers from various sources of image degradation. Several adaptive beamformers have been proposed to address this problem, including more recently proposed deep learning methods. With deep learning, adaptive beamforming is typically framed as a regression problem, where clean ground-truth physical information is used for training. Because it is difficult to know ground truth information in vivo, training data are usually simulated. However, deep networks trained on simulations can produce suboptimal in vivo image quality because of a domain shift between simulated and in vivo data. In this work, we propose a novel domain adaptation (DA) scheme to correct for domain shift by incorporating unlabeled in vivo data during training. Unlike classification tasks for which both input domains map to the same target domain, a challenge in our regression-based beamforming scenario is that domain shift exists in both the input and target data. To solve this problem, we leverage cycle-consistent generative adversarial networks to map between simulated and in vivo data in both the input and ground truth target domains. Additionally, to account for separate as well as shared features between simulations and in vivo data, we use augmented feature mapping to train domain-specific beamformers. Using various types of training data, we explore the limitations and underlying functionality of the proposed DA approach. Additionally, we compare our proposed approach to several other adaptive beamformers. Using the DA DNN beamformer, consistent in vivo image quality improvements are achieved compared to established techniques.

Evaluating Input Domain and Model Selection for Deep Network Ultrasound Beamforming

Improving ultrasound B-mode image quality remains an important area of research. Recently, there has been increased interest in using deep neural networks (DNNs) to perform beamforming to improve image quality more efficiently. Several approaches have been proposed that use different representations of channel data for network processing, including a frequency-domain approach that we previously developed. We previously assumed that the frequency domain would be more robust to varying pulse shapes. However, frequency- and time-domain implementations have not been directly compared. In addition, because our approach operates on aperture domain data as an intermediate beamforming step, a discrepancy often exists between network performance and image quality on fully reconstructed images, making model selection challenging. Here, we perform a systematic comparison of frequency- and time-domain implementations. In addition, we propose a contrast-to-noise ratio (CNR)-based regularization to address previous challenges with model selection. Training channel data were generated from simulated anechoic cysts. Test channel data were generated from simulated anechoic cysts with and without varied pulse shapes, in addition to physical phantom and in vivo data. We demonstrate that simplified time-domain implementations are more robust than we previously assumed, especially when using phase preserving data representations. Specifically, 0.39- and 0.36-dB median improvements in in vivo CNR compared to DAS were achieved with frequency- and time-domain implementations, respectively. We also demonstrate that CNR regularization improves the correlation between training validation loss and simulated CNR by 0.83 and between simulated and in vivo CNR by 0.35 compared to DNNs trained without CNR regularization.

Ultrasound deep beamforming using a multiconstrained hybrid generative adversarial network

Ultrasound beamforming is a principal factor in high-quality ultrasound imaging. The conventional delay-and-sum (DAS) beamformer generates images with high computational speed but low spatial resolution; thus, many adaptive beamforming methods have been introduced to improve image qualities. However, these adaptive beamforming methods suffer from high computational complexity, which limits their practical applications. Hence, an advanced beamformer that can overcome spatiotemporal resolution bottlenecks is eagerly awaited. In this paper, we propose a novel deep-learning-based algorithm, called the multiconstrained hybrid generative adversarial network (MC-HGAN) beamformer that rapidly achieves high-quality ultrasound imaging. The MC-HGAN beamformer directly establishes a one-shot mapping between the radio frequency signals and the reconstructed ultrasound images through a hybrid generative adversarial network (GAN) model. Through two specific branches, the hybrid GAN model extracts both radio frequency-based and image-based features and integrates them through a fusion module. We also introduce a multiconstrained training strategy to provide comprehensive guidance for the network by invoking intermediates to co-constrain the training process. Moreover, our beamformer is designed to adapt to various ultrasonic emission modes, which improves its generalizability for clinical applications. We conducted experiments on a variety of datasets scanned by line-scan and plane wave emission modes and evaluated the results with both similarity-based and ultrasound-specific metrics. The comparisons demonstrate that the MC-HGAN beamformer generates ultrasound images whose quality is higher than that of images generated by other deep learning-based methods and shows very high robustness in different clinical datasets. This technology also shows great potential in real-time imaging.

Image Quality Enhancement Using a Deep Neural Network for Plane Wave Medical Ultrasound Imaging

Plane wave imaging (PWI), a typical ultrafast medical ultrasound imaging mode, adopts single plane wave emission without focusing to achieve a high frame rate. However, the imaging quality is severely degraded in comparison with the commonly used focused line scan mode. Conventional adaptive beamformers can improve imaging quality at the cost of additional computation. In this article, we propose to use a deep neural network (DNN) to enhance the performance of PWI while maintaining a high frame rate. In particular, the PWI response from a single point target is used as the network input, while the focused scan response from the same point serves as the desired output, which is the main contribution of this method. To evaluate the performance of the proposed method, simulations, phantom experiments and in vivo studies are conducted. The delay-and-sum (DAS), the coherence factor (CF), a previously proposed deep learning-based method and the DAS with focused scan are used for comparison. Numerical metrics, including the contrast ratio (CR), the contrast-to-noise ratio (CNR), and the speckle signal-to-noise ratio (sSNR), are used to quantify the performance. The results indicate that the proposed method can achieve superior resolution and contrast performance. Specifically, the proposed method performs better than the DAS in all metrics. Although the CF provides a higher CR, its CNR and sSNR are much lower than those of the proposed method. The overall performance is also better than that of the previous deep learning method and at the same level with focused scan performance. Additionally, in comparison with the DAS, the proposed method requires little additional computation, which ensures high temporal resolution. These results validate that the proposed method can achieve a high imaging quality while maintaining the high frame rate associated with PWI.

Handheld Ultrasound Video High-Quality Reconstruction Using a Low-Rank Representation Multipathway Generative Adversarial Network

Recently, the use of portable equipment has attracted much attention in the medical ultrasound field. Handheld ultrasound devices have great potential for improving the convenience of diagnosis, but noise-induced artifacts and low resolution limit their application. To enhance the video quality of handheld ultrasound devices, we propose a low-rank representation multipathway generative adversarial network (LRR MPGAN) with a cascade training strategy. This method can directly generate sequential, high-quality ultrasound video with clear tissue structures and details. In the cascade training process, the network is first trained with plane wave (PW) single-/multiangle video pairs to capture dynamic information and then fine-tuned with handheld/high-end image pairs to extract high-quality single-frame information. In the proposed GAN structure, a multipathway generator is applied to implement the cascade training strategy, which can simultaneously extract dynamic information and synthesize multiframe features. The LRR decomposition channel approach guarantees the fine reconstruction of both global features and local details. In addition, a novel ultrasound loss is added to the conventional mean square error (MSE) loss to acquire ultrasound-specific perceptual features. A comprehensive evaluation is conducted in the experiments, and the results confirm that the proposed method can effectively reconstruct high-quality ultrasound videos for handheld devices. With the aid of the proposed method, handheld ultrasound devices can be used to obtain convincing and convenient diagnoses.

Reconstruction for Diverging-Wave Imaging Using Deep Convolutional Neural Networks

In recent years, diverging wave (DW) ultrasound imaging has become a very promising methodology for cardiovascular imaging due to its high temporal resolution. However, if they are limited in number, DW transmits provide lower image quality compared with classical focused schemes. A conventional reconstruction approach consists in summing series of ultrasound signals coherently, at the expense of frame rate, data volume, and computation time. To deal with this limitation, we propose a convolutional neural network (CNN) architecture, Inception for DW Network (IDNet), for high-quality reconstruction of DW ultrasound images using a small number of transmissions. In order to cope with the specificities induced by the sectorial geometry associated with DW imaging, we adopted the inception model composed of the concatenation of multiscale convolution kernels. Incorporating inception modules aims at capturing different image features with multiscale receptive fields. A mapping between low-quality images and corresponding high-quality compounded reconstruction was learned by training the network using in vitro and in vivo samples. The performance of the proposed approach was evaluated in terms of contrast ratio (CR), contrast-to-noise ratio (CNR), and lateral resolution (LR), and compared with standard compounding method and conventional CNN methods. The results demonstrated that our method could produce high-quality images using only 3 DWs, yielding an image quality equivalent to that obtained with compounding of 31 DWs and outperforming more conventional CNN architectures in terms of complexity, inference time, and image quality.

Deep Learning to Obtain Simultaneous Image and Segmentation Outputs From a Single Input of Raw Ultrasound Channel Data

Single plane wave transmissions are promising for automated imaging tasks requiring high ultrasound frame rates over an extended field of view. However, a single plane wave insonification typically produces suboptimal image quality. To address this limitation, we are exploring the use of deep neural networks (DNNs) as an alternative to delay-and-sum (DAS) beamforming. The objectives of this work are to obtain information directly from raw channel data and to simultaneously generate both a segmentation map for automated ultrasound tasks and a corresponding ultrasound B-mode image for interpretable supervision of the automation. We focus on visualizing and segmenting anechoic targets surrounded by tissue and ignoring or deemphasizing less important surrounding structures. DNNs trained with Field II simulations were tested with simulated, experimental phantom, and in vivo data sets that were not included during training. With unfocused input channel data (i.e., prior to the application of receive time delays), simulated, experimental phantom, and in vivo test data sets achieved mean ± standard deviation Dice similarity coefficients of 0.92 ± 0.13, 0.92 ± 0.03, and 0.77 ± 0.07, respectively, and generalized contrast-to-noise ratios (gCNRs) of 0.95 ± 0.08, 0.93 ± 0.08, and 0.75 ± 0.14, respectively. With subaperture beamformed channel data and a modification to the input layer of the DNN architecture to accept these data, the fidelity of image reconstruction increased (e.g., mean gCNR of multiple acquisitions of two in vivo breast cysts ranged 0.89-0.96), but DNN display frame rates were reduced from 395 to 287 Hz. Overall, the DNNs successfully translated feature representations learned from simulated data to phantom and in vivo data, which is promising for this novel approach to simultaneous ultrasound image formation and segmentation.

Estimation of Ultrasound Echogenicity Map from B-Mode Images Using Convolutional Neural Network

In ultrasound B-mode imaging, speckle noises decrease the accuracy of estimation of tissue echogenicity of imaged targets from the amplitude of the echo signals. In addition, since the granular size of the speckle pattern is affected by the point spread function (PSF) of the imaging system, the resolution of B-mode image remains limited, and the boundaries of tissue structures often become blurred. This study proposed a convolutional neural network (CNN) to remove speckle noises together with improvement of image spatial resolution to reconstruct ultrasound tissue echogenicity map. The CNN model is trained using in silico simulation dataset and tested with experimentally acquired images. Results indicate that the proposed CNN method can effectively eliminate the speckle noises in the background of the B-mode images while retaining the contours and edges of the tissue structures. The contrast and the contrast-to-noise ratio of the reconstructed echogenicity map increased from 0.22/2.72 to 0.33/44.14, and the lateral and axial resolutions also improved from 5.9/2.4 to 2.9/2.0, respectively. Compared with other post-processing filtering methods, the proposed CNN method provides better approximation to the original tissue echogenicity by completely removing speckle noises and improving the image resolution together with the capability for real-time implementation.

Adaptive and Compressive Beamforming Using Deep Learning for Medical Ultrasound

In ultrasound (US) imaging, various types of adaptive beamforming techniques have been investigated to improve the resolution and the contrast-to-noise ratio of the delay and sum (DAS) beamformers. Unfortunately, the performance of these adaptive beamforming approaches degrades when the underlying model is not sufficiently accurate and the number of channels decreases. To address this problem, here, we propose a deep-learning-based beamformer to generate significantly improved images over widely varying measurement conditions and channel subsampling patterns. In particular, our deep neural network is designed to directly process full or subsampled radio frequency (RF) data acquired at various subsampling rates and detector configurations so that it can generate high-quality US images using a single beamformer. The origin of such input-dependent adaptivity is also theoretically analyzed. Experimental results using the B-mode focused US confirm the efficacy of the proposed methods.

A Hierarchical Neural Network for Sleep Stage Classification Based on Comprehensive Feature Learning and Multi-Flow Sequence Learning

Automatic sleep staging methods usually extract hand-crafted features or network trained features from signals recorded by polysomnography (PSG), and then estimate the stages by various classifiers. In this study, we propose a classification approach based on a hierarchical neural network to process multi-channel PSG signals for improving the performance of automatic five-class sleep staging. The proposed hierarchical network contains two stages: comprehensive feature learning stage and sequence learning stage. The first stage is used to obtain the feature matrix by fusing the hand-crafted features and network trained features. A multi-flow recurrent neural network (RNN) as the second stage is utilized to fully learn temporal information between sleep epochs and fine-tune the parameters in the first stage. The proposed model was evaluated by 147 full night recordings in a public sleep database, the Montreal Archive of Sleep Studies (MASS). The proposed approach can achieve the overall accuracy of 0.878, and the F1-score is 0.818. The results show that the approach can achieve better performance compared to the state-of-the-art methods. Ablation experiment and model analysis proved the effectiveness of different components of the proposed model. The proposed approach allows automatic sleep stage classification by multi-channel PSG signals with different criteria standards, signal characteristics, and epoch divisions, and it has the potential to exploit sleep information comprehensively.

Image Quality Improvement of Hand-Held Ultrasound Devices With a Two-Stage Generative Adversarial Network

As a widely used imaging modality in the medical field, ultrasound has been applied in community medicine, rural medicine, and even telemedicine in recent years. Therefore, the development of portable ultrasound devices has become a popular research topic. However, the limited size of portable ultrasound devices usually degrades the imaging quality, which reduces the diagnostic reliability. To overcome hardware limitations and improve the image quality of portable ultrasound devices, we propose a novel generative adversarial network (GAN) model to achieve mapping between low-quality ultrasound images and corresponding high-quality images. In contrast to the traditional GAN method, our two-stage GAN that cascades a U-Net network prior to the generator as a front end is built to reconstruct the tissue structure, details, and speckle of the reconstructed image. In the training process, an ultrasound plane-wave imaging (PWI) data-based transfer learning method is introduced to facilitate convergence and to eliminate the influence of deformation caused by respiratory activities during data pair acquisition. A gradual tuning strategy is adopted to obtain better results by the PWI transfer learning process. In addition, a comprehensive loss function is presented to combine texture, structure, and perceptual features. Experiments are conducted using simulated, phantom, and clinical data. Our proposed method is compared to four other algorithms, including traditional gray-level-based methods and learning-based methods. The results confirm that the proposed approach obtains the optimum solution for improving quality and offering useful diagnostic information for portable ultrasound images. This technology is of great significance for providing universal medical care.