Deep learning-based multimodal fusion network for segmentation and classification of breast cancers using B-mode and elastography ultrasound images

Ultrasonography is one of the key medical imaging modalities for evaluating breast lesions. For differentiating benign from malignant lesions, computer-aided diagnosis (CAD) systems have greatly assisted radiologists by automatically segmenting and identifying features of lesions. Here, we present deep learning (DL)-based methods to segment the lesions and then classify benign from malignant, utilizing both B-mode and strain elastography (SE-mode) images. We propose a weighted multimodal U-Net (W-MM-U-Net) model for segmenting lesions where optimum weight is assigned on different imaging modalities using a weighted-skip connection method to emphasize its importance. We design a multimodal fusion framework (MFF) on cropped B-mode and SE-mode ultrasound (US) lesion images to classify benign and malignant lesions. The MFF consists of an integrated feature network (IFN) and a decision network (DN). Unlike other recent fusion methods, the proposed MFF method can simultaneously learn complementary information from convolutional neural networks (CNNs) trained using B-mode and SE-mode US images. The features from the CNNs are ensembled using the multimodal EmbraceNet model and DN classifies the images using those features. The experimental results (sensitivity of 100 ± 0.00% and specificity of 94.28 ± 7.00%) on the real-world clinical data showed that the proposed method outperforms the existing single- and multimodal methods. The proposed method predicts seven benign patients as benign three times out of five trials and six malignant patients as malignant five out of five trials. The proposed method would potentially enhance the classification accuracy of radiologists for breast cancer detection in US images.

Bi-Modal Transfer Learning for Classifying Breast Cancers via Combined B-Mode and Ultrasound Strain Imaging

Although accurate detection of breast cancer still poses significant challenges, deep learning (DL) can support more accurate image interpretation. In this study, we develop a highly robust DL model based on combined B-mode ultrasound (B-mode) and strain elastography ultrasound (SE) images for classifying benign and malignant breast tumors. This study retrospectively included 85 patients, including 42 with benign lesions and 43 with malignancies, all confirmed by biopsy. Two deep neural network models, AlexNet and ResNet, were separately trained on combined 205 B-mode and 205 SE images (80% for training and 20% for validation) from 67 patients with benign and malignant lesions. These two models were then configured to work as an ensemble using both image-wise and layer-wise and tested on a dataset of 56 images from the remaining 18 patients. The ensemble model captures the diverse features present in the B-mode and SE images and also combines semantic features from AlexNet and ResNet models to classify the benign from the malignant tumors. The experimental results demonstrate that the accuracy of the proposed ensemble model is 90%, which is better than the individual models and the model trained using B-mode or SE images alone. Moreover, some patients that were misclassified by the traditional methods were correctly classified by the proposed ensemble method. The proposed ensemble DL model will enable radiologists to achieve superior detection efficiency owing to enhance classification accuracy for breast cancers in ultrasound (US) images.

Multiparametric Photoacoustic Analysis of Human Thyroid Cancers In Vivo

Thyroid cancer is one of the most common cancers, with a global increase in incidence rate for both genders. Ultrasound-guided fine-needle aspiration is the current gold standard to diagnose thyroid cancers, but the results are inaccurate, leading to repeated biopsies and unnecessary surgeries. To reduce the number of unnecessary biopsies, we explored the use of multiparametric photoacoustic (PA) analysis in combination with the American Thyroid Association (ATA) Guideline (ATAP). In this study, we performed in vivo multispectral PA imaging on thyroid nodules from 52 patients, comprising 23 papillary thyroid cancer (PTC) and 29 benign cases. From the multispectral PA data, we calculated hemoglobin oxygen saturation level in the nodule area, then classified the PTC and benign nodules with multiparametric analysis. Statistical analyses showed that this multiparametric analysis of multispectral PA responses could classify PTC nodules. Combining the photoacoustically indicated probability of PTC and the ATAP led to a new scoring method that achieved a sensitivity of 83% and a specificity of 93%. This study is the first multiparametric analysis of multispectral PA data of thyroid nodules with statistical significance. As a proof of concept, the results show that the proposed new ATAP scoring can help physicians examine thyroid nodules for fine-needle aspiration biopsy, thus reducing unnecessary biopsies.

Radiological Society of North America/Quantitative Imaging Biomarker Alliance Shear Wave Speed Bias Quantification in Elastic and Viscoelastic Phantoms

OBJECTIVES

To quantify the bias of shear wave speed (SWS) measurements between different commercial ultrasonic shear elasticity systems and a magnetic resonance elastography (MRE) system in elastic and viscoelastic phantoms.

METHODS

Two elastic phantoms, representing healthy through fibrotic liver, were measured with 5 different ultrasound platforms, and 3 viscoelastic phantoms, representing healthy through fibrotic liver tissue, were measured with 12 different ultrasound platforms. Measurements were performed with different systems at different sites, at 3 focal depths, and with different appraisers. The SWS bias across the systems was quantified as a function of the system, site, focal depth, and appraiser. A single MRE research system was also used to characterize these phantoms using discrete frequencies from 60 to 500 Hz.

RESULTS

The SWS from different systems had mean difference 95% confidence intervals of ±0.145 m/s (±9.6%) across both elastic phantoms and ± 0.340 m/s (±15.3%) across the viscoelastic phantoms. The focal depth and appraiser were less significant sources of SWS variability than the system and site. Magnetic resonance elastography best matched the ultrasonic SWS in the viscoelastic phantoms using a 140 Hz source but had a - 0.27 ± 0.027-m/s (-12.2% ± 1.2%) bias when using the clinically implemented 60-Hz vibration source.

CONCLUSIONS

Shear wave speed reconstruction across different manufacturer systems is more consistent in elastic than viscoelastic phantoms, with a mean difference bias of < ±10% in all cases. Magnetic resonance elastographic measurements in the elastic and viscoelastic phantoms best match the ultrasound systems with a 140-Hz excitation but have a significant negative bias operating at 60 Hz. This study establishes a foundation for meaningful comparison of SWS measurements made with different platforms.

Nonlinear pth root spectral magnitude scaling beamforming for clinical photoacoustic and ultrasound imaging

A recently introduced nonlinear pth root delay-and-sum (NL-p-DAS) beamforming (BF) technique for ultrasound (US) and photoacoustic (PA) imaging, achieving better spatial and contrast resolution compared to a conventional delay and sum (DAS) technique. While the method is advantageous for better resolution, it suffers from grainy speckles and dark areas in the image mainly due to the interference of non-sinusoidal functions. In this Letter, we introduce a modified NL-p-DAS technique called nonlinear pth root spectral magnitude scaling (NL-p-SMS), which performs the pth root on the spectral magnitude instead of the temporal amplitude. We evaluated the US and PA images of NL-p-SMS against those of NL-p-DAS by comparing the axial and lateral line profiles, contrasts, and contrast-to-noise ratios (CNRs) in both phantom and in vivo imaging studies with various p values. As a result, we found that the NL-p-SMS has better axial resolution and CNR than the NL-p-DAS, and reduces the grainy speckles and dark area artifacts. We believe that, with this enhanced performance, our proposed approach could be an advancement compared to the existing nonlinear BF algorithms.

3D PHOVIS: 3D photoacoustic visualization studio

Photoacoustic (PA) imaging (or optoacoustic imaging) is a novel biomedical imaging method in biological and medical research. This modality performs morphological, functional, and molecular imaging with and without labels in both microscopic and deep tissue imaging domains. A variety of innovations have enhanced 3D PA imaging performance and thus has opened new opportunities in preclinical and clinical imaging. However, the 3D visualization tools for PA images remains a challenge. There are several commercially available software packages to visualize the generated 3D PA images. They are generally expensive, and their features are not optimized for 3D visualization of PA images. Here, we demonstrate a specialized 3D visualization software package, namely 3D Photoacoustic Visualization Studio (3D PHOVIS), specifically targeting photoacoustic data, image, and visualization processes. To support the research environment for visualization and fast processing, we incorporated 3D PHOVIS onto the MATLAB with graphical user interface and developed multi-core graphics processing unit modules for fast processing. The 3D PHOVIS includes following modules: (1) a mosaic volume generator, (2) a scan converter for optical scanning photoacoustic microscopy, (3) a skin profile estimator and depth encoder, (4) a multiplanar viewer with a navigation map, and (5) a volume renderer with a movie maker. This paper discusses the algorithms present in the software package and demonstrates their functions. In addition, the applicability of this software to ultrasound imaging and optical coherence tomography is also investigated. User manuals and application files for 3D PHOVIS are available for free on the website (www.boa-lab.com). Core functions of 3D PHOVIS are developed as a result of a summer class at POSTECH, "High-Performance Algorithm in CPU/GPU/DSP, and Computer Architecture." We believe our 3D PHOVIS provides a unique tool to PA imaging researchers, expedites its growth, and attracts broad interests in a wide range of studies.

Real-time delay-multiply-and-sum beamforming with coherence factor for in vivo clinical photoacoustic imaging of humans

In the clinical photoacoustic (PA) imaging, ultrasound (US) array transducers are typically used to provide B-mode images in real-time. To form a B-mode image, delay-and-sum (DAS) beamforming algorithm is the most commonly used algorithm because of its ease of implementation. However, this algorithm suffers from low image resolution and low contrast drawbacks. To address this issue, delay-multiply-and-sum (DMAS) beamforming algorithm has been developed to provide enhanced image quality with higher contrast, and narrower main lobe compared but has limitations on the imaging speed for clinical applications. In this paper, we present an enhanced real-time DMAS algorithm with modified coherence factor (CF) for clinical PA imaging of humans in vivo. Our algorithm improves the lateral resolution and signal-to-noise ratio (SNR) of original DMAS beamformer by suppressing the background noise and side lobes using the coherence of received signals. We optimized the computations of the proposed DMAS with CF (DMAS-CF) to achieve real-time frame rate imaging on a graphics processing unit (GPU). To evaluate the proposed algorithm, we implemented DAS and DMAS with/without CF on a clinical US/PA imaging system and quantitatively assessed their processing speed and image quality. The processing time to reconstruct one B-mode image using DAS, DAS with CF (DAS-CF), DMAS, and DMAS-CF algorithms was 7.5, 7.6, 11.1, and 11.3 ms, respectively, all achieving the real-time imaging frame rate. In terms of the image quality, the proposed DMAS-CF algorithm improved the lateral resolution and SNR by 55.4% and 93.6 dB, respectively, compared to the DAS algorithm in the phantom imaging experiments. We believe the proposed DMAS-CF algorithm and its real-time implementation contributes significantly to the improvement of imaging quality of clinical US/PA imaging system.

A Novel 2-D Synthetic Aperture Focusing Technique for Acoustic-Resolution Photoacoustic Microscopy

Acoustic-resolution photoacoustic microscopy (AR-PAM) is a promising technology for vascular or tumor-targeted molecular imaging. Unique advantages of AR-PM are its non-invasive, non-ionizing real-time, and deeper imaging depth. AR-PAM typically uses an ultrasound transducer with a high acoustic numerical aperture (NA) to enable deeper imaging depth. While high NA achieves good lateral resolution in the focal plane but significantly degrades the lateral resolution in the out-of-focus region. Synthetic aperture focusing technique (SAFT) has been introduced to overcome this out-of-focus degradation by synthesizing the correlated signals. Several 2-D SAFTs have been also reported to improve degraded resolution in all directions. However, the resolution enhancement of the previously reported 2-D SAFTs are suboptimal and are not equivalent to the 1-D SAFT performance under an ideal condition with the sample orientation perpendicular to the synthetic aperture direction. In this paper, we present a new 2-D SAFT called 2-D directional SAFT that improves the lateral resolution significantly and we compare our results against 1-D SAFT under ideal condition. We applied this algorithm to phantom and in vivo images to show the improvement in image quality. We also implement this algorithm in a graphical processing unit to achieve high performance to show the practicality of implementing this new algorithm in a system.

In Vivo Photoacoustic Imaging of Anterior Ocular Vasculature: A Random Sample Consensus Approach

Visualizing ocular vasculature is important in clinical ophthalmology because ocular circulation abnormalities are early signs of ocular diseases. Photoacoustic microscopy (PAM) images the ocular vasculature without using exogenous contrast agents, avoiding associated side effects. Moreover, 3D PAM images can be useful in understanding vessel-related eye disease. However, the complex structure of the multi-layered vessels still present challenges in evaluating ocular vasculature. In this study, we demonstrate a new method to evaluate blood circulation in the eye by combining in vivo PAM imaging and an ocular surface estimation method based on a machine learning algorithm: a random sample consensus algorithm. By using the developed estimation method, we were able to visualize the PA ocular vascular image intuitively and demonstrate layer-by-layer analysis of injured ocular vasculature. We believe that our method can provide more accurate evaluations of the eye circulation in ophthalmic applications.

Research interface on a programmable ultrasound scanner

MOTIVATION

Commercial ultrasound machines in the past did not provide the ultrasound researchers access to raw ultrasound data. Lack of this ability has impeded evaluation and clinical testing of novel ultrasound algorithms and applications.

OBJECTIVES

Recently, we developed a flexible ultrasound back-end where all the processing for the conventional ultrasound modes, such as B, M, color flow and spectral Doppler, was performed in software. The back-end has been incorporated into a commercial ultrasound machine, the Hitachi HiVision 5500. The goal of this work is to develop an ultrasound research interface on the back-end for acquiring raw ultrasound data from the machine.

METHODS

The research interface has been designed as a software module on the ultrasound back-end. To increase the amount of raw ultrasound data that can be spooled in the limited memory available on the back-end, we have developed a method that can losslessly compress the ultrasound data in real time.

RESULTS AND DISCUSSION

The raw ultrasound data could be obtained in any conventional ultrasound mode, including duplex and triplex modes. Furthermore, use of the research interface does not decrease the frame rate or otherwise affect the clinical usability of the machine. The lossless compression of the ultrasound data in real time can increase the amount of data spooled by approximately 2.3 times, thus allowing more than 6s of raw ultrasound data to be acquired in all the modes. The interface has been used not only for early testing of new ideas with in vitro data from phantoms, but also for acquiring in vivo data for fine-tuning ultrasound applications and conducting clinical studies. We present several examples of how newer ultrasound applications, such as elastography, vibration imaging and 3D imaging, have benefited from this research interface. Since the research interface is entirely implemented in software, it can be deployed on existing HiVision 5500 ultrasound machines and may be easily upgraded in the future.

CONCLUSIONS

The developed research interface can aid researchers in the rapid testing and clinical evaluation of new ultrasound algorithms and applications. Additionally, we believe that our approach would be applicable to designing research interfaces on other ultrasound machines.

New multi-volume rendering technique for three-dimensional power Doppler imaging

In this paper, we present a new multi-volume rendering technique (i.e., progressive fusion) to combine 3D anatomical structures from B-mode imaging and flow information from power Doppler imaging. A post-fusion technique, in which B-mode and power Doppler volumes are independently rendered and then fused based on alpha blending, is typically used in 3D power Doppler imaging. However, it has limitations in preserving the spatial relationship (i.e., depth order) between tissue structure and vasculature since they are rendered independently and then merged. With the proposed progressive fusion, B-mode and power Doppler volumes are composited together while rendering by sharing the opacity values. After compositing, two rendered frames are blended by utilizing a 2D color lookup table designed to fuse two properties (i.e., tissues and blood flows). We have evaluated the progressive-fusion multi-volume rendering method with the phantom and in vivo data acquired using a commercial ultrasound machine (EUB-8500, Hitachi Medical Corporation, Japan) with a 3.5 MHz mechanical probe. From the preliminary study, we have found that the new progressive-fusion method can better retain and display the spatial relationship between tissue structure, vasculature and their corresponding depth order.

Tomosynthesis-based localization of radioactive seeds in prostate brachytherapy

Accurately assessing the quality of prostate brachytherapy intraoperatively would be valuable for improved clinical outcome by ensuring the delivery of a prescribed tumoricidal radiation dose to the entire prostate gland. One necessary step towards this goal is the robust and rapid localization of implanted seeds. Several methods have been developed to locate seeds from x-ray projection images, but they fail to detect completely-overlapping seeds, thus necessitating manual intervention. To overcome this limitation, we have developed a new method where (1) a three-dimensional volume is reconstructed from x-ray projection images using a brachytherapy-specific tomosynthesis reconstruction algorithm with built-in blur compensation and (2) the seeds are located in this reconstructed volume. In contrast to other projection-based methods, our method can detect completely overlapping seeds. Our simulation results indicate that we can locate all implanted seeds in the prostate using a tomosynthesis angle of 30 degrees and seven projection images. The mean localization error is 1.27 mm for a case with 100 seeds. We have also tested our method using a prostate phantom with 61 implanted seeds and succeeded in locating all seeds automatically. We believe this new method can be useful for the intraoperative quality assessment of prostate brachytherapy in the future.

Ultrasound Color-Flow Imaging on a Programmable System

Color-flow imaging is a well-established ultrasound mode and very valuable for visualizing in real time the distribution of blood flow in a specific region of interest. However, it is computationally quite expensive. To meet the large computational need in color-flow imaging, most ultrasound systems have been designed using fixed-function hardware. In this paper, we present a system where all the color-flow processing is supported on a programmable platform. About 95% of the processing modules were programmed in C language. On a single processor, we were able to achieve 7.9 frames/s, when the input data consist of 192 x 512 x 8 (ensemble size) samples for color flow and 384 x 512 for B mode and the output image size is 600 x 420. Additional processors can be added to handle more input data and/or support higher frame rates. Our results demonstrate that a programmable ultrasound system can provide the same functionality for clinical use as conventional ultrasound systems. However, it is more flexible and efficient due to its programmability.

Fast adaptive unsharp masking with programmable mediaprocessors

Unsharp masking is a widely used image-enhancement method in medical imaging. Hardware-based solutions can be developed to support high computational demand for unsharp masking, but they suffer from limited flexibility. Software solutions can easily incorporate new features and modify key parameters, such as filtering kernel size, but they have not been able to meet the fast computing requirement. Modern programmable mediaprocessors can meet both fast computing and flexibility requirements, which will benefit medical image computing. In this article, we present fast adaptive unsharp masking on two leading mediaprocessors or high-end digital signal processors, Hitachi/Equator Technologies MAP-CA and Texas Instruments TMS320C64x. For a 2k x 2k 16-bit image, our adaptive unsharp masking with a 201 x 201 boxcar kernel takes 225 ms on a 300-MHz MAP-CA and 74 ms on a 600-MHz TMS320C64x. This fast unsharp masking enables technologists and/or physicians to adjust parameters interactively for optimal quality assurance and image viewing.

Adaptive clutter filtering for ultrasound color flow imaging

In this article, we present an adaptive clutter rejection method for selecting different clutter filters in ultrasound color flow imaging. A single clutter filter is typically used to reject the clutter. Because the clutter characteristics vary in both space and time, the single clutter filter approach has difficulty in providing optimum clutter rejection in ultrasound images. To achieve more accurate velocity estimation, we have developed a method to select a clutter filter adaptively at each location in an image from a set of predefined filters. Selection criteria have been developed based on the underlying clutter characteristics and the properties of various filters (e.g., minimum-phase finite impulse response, projection-initialized infinite impulse response and polynomial regression). We have incorporated our adaptive clutter rejection method in an ultrasound system. We have found that our adaptive method can reduce the mean absolute error between the estimated and true flow velocities significantly compared with the conventional methods, in which a single clutter filter is used throughout the entire image. With in vivo abdominal data, we obtained an average gain of 5.0 dB in signal-to-clutter ratio (SCR), compared with the conventional method. These preliminary results indicate that the proposed adaptive method could improve the accuracy of flow velocity estimation in ultrasound color flow imaging through the improvement in SCR and the reduction in bias.

A single mediaprocessor-based programmable ultrasound system

We have developed a programmable ultrasound imaging system using a single commercially available mediaprocessor. We have efficiently mapped all of the necessary B-mode processing algorithms on the underlying processor architecture, including envelope detection, dynamic range compression, lateral and axial filtering, persistence processing, and scan conversion. Our system can handle varying specifications ranging from 128 vectors and 512 samples per vector to more than 256 vectors and 1024 samples per vector. For an image size of 330 vectors and 512 samples per vector, it can process 30 frames per second using a 300-MHz MAP-CA mediaprocessor from Hitachi/Equator Technologies. This programmable ultrasound machine will not only offer significant advantages in terms of low cost, portability, scalability, and reduced development time, but also provide a flexible platform for developing and deploying new clinical applications to aid the clinicians and improve the quality of healthcare to patients.