Model-Based Learning for Accelerated, Limited-View 3-D Photoacoustic Tomography

LV-GAN: A deep learning approach for limited-view optoacoustic imaging based on hybrid datasets

The optoacoustic imaging (OAI) methods are rapidly evolving for resolving optical contrast in medical imaging applications. In practice, measurement strategies are commonly implemented under limited-view conditions due to oversized image objectives or system design limitations. Data acquired by limited-view detection may impart artifacts and distortions in reconstructed optoacoustic (OA) images. We propose a hybrid data-driven deep learning approach based on generative adversarial network (GAN), termed as LV-GAN, to efficiently recover high quality images from limited-view OA images. Trained on both simulation and experiment data, LV-GAN is found capable of achieving high recovery accuracy even under limited detection angles less than 60^° . The feasibility of LV-GAN for artifact removal in biological applications was validated by ex vivo experiments based on two different OAI systems, suggesting high potential of a ubiquitous use of LV-GAN to optimize image quality or system design for different scanners and application scenarios.

On Learned Operator Correction in Inverse Problems

We discuss the possibility of learning a data-driven explicit model correction for inverse problems and whether such a model correction can be used within a variational framework to obtain regularized reconstructions. This paper discusses the conceptual difficulty of learning such a forward model correction and proceeds to present a possible solution as a forward-adjoint correction that explicitly corrects in both data and solution spaces. We then derive conditions under which solutions to the variational problem with a learned correction converge to solutions obtained with the correct operator. The proposed approach is evaluated on an application to limited view photoacoustic tomography and compared to the established framework of the Bayesian approximation error method.

Another decade of photoacoustic imaging

Photoacoustic imaging - a hybrid biomedical imaging modality finding its way to clinical practices. Although the photoacoustic phenomenon was known more than a century back, only in the last two decades it has been widely researched and used for biomedical imaging applications. In this review we focus on the development and progress of the technology in the last decade (2010-2020). From becoming more and more user friendly, cheaper in cost, portable in size, photoacoustic imaging promises a wide range of applications, if translated to clinic. The growth of photoacoustic community is steady, and with several new directions researchers are exploring, it is inevitable that photoacoustic imaging will one day establish itself as a regular imaging system in the clinical practices.

Sound Out the Deep Colors: Photoacoustic Molecular Imaging at New Depths

Photoacoustic tomography (PAT) has become increasingly popular for molecular imaging due to its unique optical absorption contrast, high spatial resolution, deep imaging depth, and high imaging speed. Yet, the strong optical attenuation of biological tissues has traditionally prevented PAT from penetrating more than a few centimeters and limited its application for studying deeply seated targets. A variety of PAT technologies have been developed to extend the imaging depth, including employing deep-penetrating microwaves and X-ray photons as excitation sources, delivering the light to the inside of the organ, reshaping the light wavefront to better focus into scattering medium, as well as improving the sensitivity of ultrasonic transducers. At the same time, novel optical fluence mapping algorithms and image reconstruction methods have been developed to improve the quantitative accuracy of PAT, which is crucial to recover weak molecular signals at larger depths. The development of highly-absorbing near-infrared PA molecular probes has also flourished to provide high sensitivity and specificity in studying cellular processes. This review aims to introduce the recent developments in deep PA molecular imaging, including novel imaging systems, image processing methods and molecular probes, as well as their representative biomedical applications. Existing challenges and future directions are also discussed.

A Deep Learning Approach to Photoacoustic Wavefront Localization in Deep-Tissue Medium

Optical photons undergo strong scattering when propagating beyond 1-mm deep inside biological tissue. Finding the origin of these diffused optical wavefronts is a challenging task. Breaking through the optical diffusion limit, photoacoustic (PA) imaging (PAI) provides high-resolution and label-free images of human vasculature with high contrast due to the optical absorption of hemoglobin. In real-time PAI, an ultrasound transducer array detects PA signals, and B-mode images are formed by delay-and-sum or frequency-domain beamforming. Fundamentally, the strength of a PA signal is proportional to the local optical fluence, which decreases with the increasing depth due to depth-dependent optical attenuation. This limits the visibility of deep-tissue vasculature or other light-absorbing PA targets. To address this practical challenge, an encoder-decoder convolutional neural network architecture was constructed with custom modules and trained to identify the origin of the PA wavefronts inside an optically scattering deep-tissue medium. A comprehensive ablation study provides strong evidence that each module improves the localization accuracy. The network was trained on model-based simulated PA signals produced by 16 240 blood-vessel targets subjected to both optical scattering and Gaussian noise. Test results on 4600 simulated and five experimental PA signals collected under various scattering conditions show that the network can localize the targets with a mean error less than 30 microns (standard deviation 20.9 microns) for targets below 40-mm imaging depth and 1.06 mm (standard deviation 2.68 mm) for targets at a depth between 40 and 60 mm. The proposed work has broad applications such as diffused optical wavefront shaping, circulating melanoma cell detection, and real-time vascular surgeries (e.g., deep-vein thrombosis).

Deep Neural Network-Based Sinogram Super-Resolution and Bandwidth Enhancement for Limited-Data Photoacoustic Tomography

Photoacoustic tomography (PAT) is a noninvasive imaging modality combining the benefits of optical contrast at ultrasonic resolution. Analytical reconstruction algorithms for photoacoustic (PA) signals require a large number of data points for accurate image reconstruction. However, in practical scenarios, data are collected using the limited number of transducers along with data being often corrupted with noise resulting in only qualitative images. Furthermore, the collected boundary data are band-limited due to limited bandwidth (BW) of the transducer, making the PA imaging with limited data being qualitative. In this work, a deep neural network-based model with loss function being scaled root-mean-squared error was proposed for super-resolution, denoising, as well as BW enhancement of the PA signals collected at the boundary of the domain. The proposed network has been compared with traditional as well as other popular deep-learning methods in numerical as well as experimental cases and is shown to improve the collected boundary data, in turn, providing superior quality reconstructed PA image. The improvement obtained in the Pearson correlation, structural similarity index metric, and root-mean-square error was as high as 35.62%, 33.81%, and 41.07%, respectively, for phantom cases and signal-to-noise ratio improvement in the reconstructed PA images was as high as 11.65 dB for in vivo cases compared with reconstructed image obtained using original limited BW data. Code is available at https://sites.google.com/site/sercmig/home/dnnpat.

Y-Net: Hybrid deep learning image reconstruction for photoacoustic tomography in vivo

Conventional reconstruction algorithms (e.g., delay-and-sum) used in photoacoustic imaging (PAI) provide a fast solution while many artifacts remain, especially for limited-view with ill-posed problem. In this paper, we propose a new convolutional neural network (CNN) framework Y-Net: a CNN architecture to reconstruct the initial PA pressure distribution by optimizing both raw data and beamformed images once. The network combines two encoders with one decoder path, which optimally utilizes more information from raw data and beamformed image. We compared our result with some ablation studies, and the results of the test set show better performance compared with conventional reconstruction algorithms and other deep learning method (U-Net). Both in-vitro and in-vivo experiments are used to validated our method, which still performs better than other existing methods. The proposed Y-Net architecture also has high potential in medical image reconstruction for other imaging modalities beyond PAI.

Miniature all-optical flexible forward-viewing photoacoustic endoscopy probe for surgical guidance

A miniature flexible photoacoustic endoscopy probe that provides high-resolution 3D images of vascular structures in the forward-viewing configuration is described. A planar Fabry-Perot ultrasound sensor with a -3dB bandwidth of 53 MHz located at the tip of the probe is interrogated via a flexible fiber bundle and a miniature optical relay system to realize an all-optical probe measuring 7.4 mm in outer diameter at the tip. This approach to photoacoustic endoscopy offers advantages over previous piezoelectric based distal-end scanning probes. These include a forward-viewing configuration in widefield photoacoustic tomography mode, finer spatial sampling (87 µm spatial sampling interval), and wider detection bandwidth (53 MHz) than has been achievable with conventional ultrasound detection technology and an all-optical passive imaging head for safe endoscopic use.

Multispectral Interlaced Sparse Sampling Photoacoustic Tomography

Multispectral photoacoustic tomography (PAT) is capable of resolving tissue chromophore distribution based on spectral un-mixing. It works by identifying the absorption spectrum variations from a sequence of photoacoustic images acquired at multiple illumination wavelengths. Due to multispectral acquisition, this inevitably creates a large dataset. To cut down the data volume, sparse sampling methods that reduce the number of detectors have been developed. However, image reconstruction of sparse sampling PAT is challenging because of insufficient angular coverage. During spectral un-mixing, these inaccurate reconstructions will further amplify imaging artefacts and contaminate the results. To solve this problem, we present the interlaced sparse sampling (ISS) PAT, a method that involved: 1) a novel scanning-based image acquisition scheme in which the sparse detector array rotates while switching illumination wavelength, such that a dense angular coverage could be achieved by using only a few detectors; and 2) a corresponding image reconstruction algorithm that makes use of an anatomical prior image created from the ISS strategy to guide PAT image computation. Reconstructed from the signals acquired at different wavelengths (angles), this self-generated prior image fuses multispectral and angular information, and thus has rich anatomical features and minimum artefacts. A specialized iterative imaging model that effectively incorporates this anatomical prior image into the reconstruction process is also developed. Simulation, phantom, and in vivo animal experiments showed that even under 1/6 or 1/8 sparse sampling rate, our method achieved comparable image reconstruction and spectral un-mixing results to those obtained by conventional dense sampling method.

Deep Learning for Ultrasound Localization Microscopy

By localizing microbubbles (MBs) in the vasculature, ultrasound localization microscopy (ULM) has recently been proposed, which greatly improves the spatial resolution of ultrasound (US) imaging and will be helpful for clinical diagnosis. Nevertheless, several challenges remain in fast ULM imaging. The main problems are that current localization methods used to implement fast ULM imaging, e.g., a previously reported localization method based on sparse recovery (CS-ULM), suffer from long data-processing time and exhaustive parameter tuning (optimization). To address these problems, in this paper, we propose a ULM method based on deep learning, which is achieved by using a modified sub-pixel convolutional neural network (CNN), termed as mSPCN-ULM. Simulations and in vivo experiments are performed to evaluate the performance of mSPCN-ULM. Simulation results show that even if under high-density condition (6.4 MBs/mm²), a high localization precision ( [Formula: see text] in the lateral direction and [Formula: see text] in the axial direction) and a high localization reliability (Jaccard index of 0.66) can be obtained by mSPCN-ULM, compared to CS-ULM. The in vivo experimental results indicate that with plane wave scan at a transmit center frequency of 15.625 MHz, microvessels with diameters of [Formula: see text] can be detected and adjacent microvessels with a distance of [Formula: see text] can be separated. Furthermore, when using GPU acceleration, the data-processing time of mSPCN-ULM can be shortened to ~6 sec/frame in the simulations and ~23 sec/frame in the in vivo experiments, which is 3-4 orders of magnitude faster than CS-ULM. Finally, once the network is trained, mSPCN-ULM does not need parameter tuning to implement ULM. As a result, mSPCN-ULM opens the door to implement ULM with fast data-processing speed, high imaging accuracy, short data-acquisition time, and high flexibility (robustness to parameters) characteristics.

End-to-end Res-Unet based reconstruction algorithm for photoacoustic imaging

Recently, deep neural networks have attracted great attention in photoacoustic imaging (PAI). In PAI, reconstructing the initial pressure distribution from acquired photoacoustic (PA) signals is a typically inverse problem. In this paper, an end-to-end Unet with residual blocks (Res-Unet) is designed and trained to solve the inverse problem in PAI. The performance of the proposed algorithm is explored and analyzed by comparing a recent model-resolution-based regularization algorithm (MRR) with numerical and physical phantom experiments. The improvement obtained in the reconstructed images was more than 95% in pearson correlation and 39% in peak signal-to-noise ratio in comparison to the MRR. The Res-Unet also achieved superior performance over the state-of-the-art Unet++ architecture by more than 18% in PSNR in simulation experiments.

Domain Transform Network for Photoacoustic Tomography from Limited-view and Sparsely Sampled Data

Medical image reconstruction methods based on deep learning have recently demonstrated powerful performance in photoacoustic tomography (PAT) from limited-view and sparse data. However, because most of these methods must utilize conventional linear reconstruction methods to implement signal-to-image transformations, their performance is restricted. In this paper, we propose a novel deep learning reconstruction approach that integrates appropriate data pre-processing and training strategies. The Feature Projection Network (FPnet) presented herein is designed to learn this signal-to-image transformation through data-driven learning rather than through direct use of linear reconstruction. To further improve reconstruction results, our method integrates an image post-processing network (U-net). Experiments show that the proposed method can achieve high reconstruction quality from limited-view data with sparse measurements. When employing GPU acceleration, this method can achieve a reconstruction speed of 15 frames per second.

Three-dimensional view of out-of-plane artifacts in photoacoustic imaging using a laser-integrated linear-transducer-array probe

Research on photoacoustic imaging (PAI) using a handheld integrated photoacoustic probe has been a recent focus of clinical translation of this imaging technique. One of the remaining challenges is the occurrence of out-of-plane artifacts (OPAs) in such a probe. Previously, we proposed a method to identify and remove OPAs by axially displacing the transducer array. Here we show that besides the benefit of removing OPAs from the imaging plane, the proposed method can provide a three-dimensional (3D) view of the OPAs. In this work, we present a 3D reconstruction method using axial transducer array displacement. By axially displacing the transducer array, out-of-plane absorbers can be three-dimensionally visualized at an elevation distance of up to the acquired imaging depth. Additionally, OPAs in the in-plane image are significantly reduced. We experimentally demonstrate the method with phantom and in vivo experiments using an integrated PAI probe. We also compare the method with elevational transducer array displacement and take into account the sensitivity of the transducer array in the 3D reconstruction.

Model-Based Reconstruction of Large Three-Dimensional Optoacoustic Datasets

Iterative model-based algorithms are known to enable more accurate and quantitative optoacoustic (photoacoustic) tomographic reconstructions than standard back-projection methods. However, three-dimensional (3D) model-based inversion is often hampered by high computational complexity and memory overhead. Parallel implementations on a graphics processing unit (GPU) have been shown to efficiently reduce the memory requirements by on-the-fly calculation of the actions of the optoacoustic model matrix, but the high complexity still makes these approaches impractical for large 3D optoacoustic datasets. Herein, we show that the computational complexity of 3D model-based iterative inversion can be significantly reduced by splitting the model matrix into two parts: one maximally sparse matrix containing only one entry per voxel-transducer pair and a second matrix corresponding to cyclic convolution. We further suggest reconstructing the images by multiplying the transpose of the model matrix calculated in this manner with the acquired signals, which is equivalent to using a very large regularization parameter in the iterative inversion method. The performance of these two approaches is compared to that of standard back-projection and a recently introduced GPU-based model-based method using datasets from in vivo experiments. The reconstruction time was accelerated by approximately an order of magnitude with the new iterative method, while multiplication with the transpose of the matrix is shown to be as fast as standard back-projection.

Photoacoustic imaging for surgical guidance: Principles, applications, and outlook

Minimally invasive surgeries often require complicated maneuvers and delicate hand-eye coordination and ideally would incorporate "x-ray vision" to see beyond tool tips and underneath tissues prior to making incisions. Photoacoustic imaging has the potential to offer this feature but not with ionizing x-rays. Instead, optical fibers and acoustic receivers enable photoacoustic sensing of major structures-such as blood vessels and nerves-that are otherwise hidden from view. This imaging process is initiated by transmitting laser pulses that illuminate regions of interest, causing thermal expansion and the generation of sound waves that are detectable with conventional ultrasound transducers. The recorded signals are then converted to images through the beamforming process. Photoacoustic imaging may be implemented to both target and avoid blood-rich surgical contents (and in some cases simultaneously or independently visualize optical fiber tips or metallic surgical tool tips) in order to prevent accidental injury and assist device operators during minimally invasive surgeries and interventional procedures. Novel light delivery systems, counterintuitive findings, and robotic integration methods introduced by the Photoacoustic & Ultrasonic Systems Engineering Lab are summarized in this invited Perspective, setting the foundation and rationale for the subsequent discussion of the author's views on possible future directions for this exciting frontier known as photoacoustic-guided surgery.

Toward accurate quantitative photoacoustic imaging: learning vascular blood oxygen saturation in three dimensions

SIGNIFICANCE

Two-dimensional (2-D) fully convolutional neural networks have been shown capable of producing maps of sO2 from 2-D simulated images of simple tissue models. However, their potential to produce accurate estimates in vivo is uncertain as they are limited by the 2-D nature of the training data when the problem is inherently three-dimensional (3-D), and they have not been tested with realistic images.

AIM

To demonstrate the capability of deep neural networks to process whole 3-D images and output 3-D maps of vascular sO2 from realistic tissue models/images.

APPROACH

Two separate fully convolutional neural networks were trained to produce 3-D maps of vascular blood oxygen saturation and vessel positions from multiwavelength simulated images of tissue models.

RESULTS

The mean of the absolute difference between the true mean vessel sO2 and the network output for 40 examples was 4.4% and the standard deviation was 4.5%.

CONCLUSIONS

3-D fully convolutional networks were shown capable of producing accurate sO2 maps using the full extent of spatial information contained within 3-D images generated under conditions mimicking real imaging scenarios. We demonstrate that networks can cope with some of the confounding effects present in real images such as limited-view artifacts and have the potential to produce accurate estimates in vivo.

Neural networks-based regularization for large-scale medical image reconstruction

In this paper we present a generalized Deep Learning-based approach for solving ill-posed large-scale inverse problems occuring in medical image reconstruction. Recently, Deep Learning methods using iterative neural networks (NNs) and cascaded NNs have been reported to achieve state-of-the-art results with respect to various quantitative quality measures as PSNR, NRMSE and SSIM across different imaging modalities. However, the fact that these approaches employ the application of the forward and adjoint operators repeatedly in the network architecture requires the network to process the whole images or volumes at once, which for some applications is computationally infeasible. In this work, we follow a different reconstruction strategy by strictly separating the application of the NN, the regularization of the solution and the consistency with the measured data. The regularization is given in the form of an image prior obtained by the output of a previously trained NN which is used in a Tikhonov regularization framework. By doing so, more complex and sophisticated network architectures can be used for the removal of the artefacts or noise than it is usually the case in iterative NNs. Due to the large scale of the considered problems and the resulting computational complexity of the employed networks, the priors are obtained by processing the images or volumes as patches or slices. We evaluated the method for the cases of 3D cone-beam low dose CT and undersampled 2D radial cine MRI and compared it to a total variation-minimization-based reconstruction algorithm as well as to a method with regularization based on learned overcomplete dictionaries. The proposed method outperformed all the reported methods with respect to all chosen quantitative measures and further accelerates the regularization step in the reconstruction by several orders of magnitude.

Rising role of artificial intelligence in image reconstruction for biomedical imaging

In this editorial, we review recent progress on the applications of artificial intelligence (AI) in image reconstruction for biomedical imaging. Because it abandons prior information of traditional artificial design and adopts a completely data-driven mode to obtain deeper prior information via learning, AI technology plays an increasingly important role in biomedical image reconstruction. The combination of AI technology and the biomedical image reconstruction method has become a hotspot in the field. Favoring AI, the performance of biomedical image reconstruction has been improved in terms of accuracy, resolution, imaging speed, etc. We specifically focus on how to use AI technology to improve the performance of biomedical image reconstruction, and propose possible future directions in this field.

A New Deep Learning Network for Mitigating Limited-view and Under-sampling Artifacts in Ring-shaped Photoacoustic Tomography

Photoacoustic tomography (PAT) is a hybrid technique for high-resolution imaging of optical absorption in tissue. Among various transducer arrays proposed for PAT, the ring-shaped transducer array is widely used in cross-sectional imaging applications. However, due to the high fabrication cost, most ring-shaped transducer arrays have a sparse transducer arrangement, which leads to limited-view problems and under-sampling artifacts. To address these issues, we paired conventional PAT reconstruction with deep learning, which recently achieved a breakthrough in image processing and tomographic reconstruction. In this study, we designed a convolutional neural network (CNN) called a ring-array deep learning network (RADL-net), which can eliminate limited-view and under-sampling artifacts in PAT images. The method was validated on a three-quarter ring transducer array using numerical simulation, phantom imaging, and in vivo imaging. Our results indicate that the proposed RADL-net significantly improves the quality of reconstructed images on a three-quarter ring transducer array. The method is also superior to the conventional compressed sensing (CS) algorithm.

Accelerated Photoacoustic Tomography Reconstruction via Recurrent Inference Machines

Accelerated photoacoustic tomography (PAT) reconstruction is important for real-time photoacoustic imaging (PAI) applications. PAT requires a reconstruction algorithm to reconstruct the detected photoacoustic signal in order to obtain the detected image of the tissue, which is usually an inverse problem. Different from the typical method for solving the inverse problems that defines a model and chooses an inference procedure, we propose to use the Recurrent Inference Machines (RIM) as a framework for PAT reconstruction. Our model performs an accelerated iterative reconstruction, and directly learns to solve the inverse problem in PAT using the information from a forward model that is based on k-space methods. As shown in experiments, our method achieves faster high-resolution PAT reconstruction, and outperforms another method based on deep neural network in some respects.

Limited-View and Sparse Photoacoustic Tomography for Neuroimaging with Deep Learning

Photoacoustic tomography (PAT) is a non-ionizing imaging modality capable of acquiring high contrast and resolution images of optical absorption at depths greater than traditional optical imaging techniques. Practical considerations with instrumentation and geometry limit the number of available acoustic sensors and their "view" of the imaging target, which result in image reconstruction artifacts degrading image quality. Iterative reconstruction methods can be used to reduce artifacts but are computationally expensive. In this work, we propose a novel deep learning approach termed pixel-wise deep learning (Pixel-DL) that first employs pixel-wise interpolation governed by the physics of photoacoustic wave propagation and then uses a convolution neural network to reconstruct an image. Simulated photoacoustic data from synthetic, mouse-brain, lung, and fundus vasculature phantoms were used for training and testing. Results demonstrated that Pixel-DL achieved comparable or better performance to iterative methods and consistently outperformed other CNN-based approaches for correcting artifacts. Pixel-DL is a computationally efficient approach that enables for real-time PAT rendering and improved image reconstruction quality for limited-view and sparse PAT.

Multi-Scale Learned Iterative Reconstruction

Model-based learned iterative reconstruction methods have recently been shown to outperform classical reconstruction algorithms. Applicability of these methods to large scale inverse problems is however limited by the available memory for training and extensive training times, the latter due to computationally expensive forward models. As a possible solution to these restrictions we propose a multi-scale learned iterative reconstruction scheme that computes iterates on discretisations of increasing resolution. This procedure does not only reduce memory requirements, it also considerably speeds up reconstruction and training times, but most importantly is scalable to large scale inverse problems with non-trivial forward operators, such as those that arise in many 3D tomographic applications. In particular, we propose a hybrid network that combines the multiscale iterative approach with a particularly expressive network architecture which in combination exhibits excellent scalability in 3D. Applicability of the algorithm is demonstrated for 3D cone beam computed tomography from real measurement data of an organic phantom. Additionally, we examine scalability and reconstruction quality in comparison to established learned reconstruction methods in two dimensions for low dose computed tomography on human phantoms.

A generative adversarial network for artifact removal in photoacoustic computed tomography with a linear-array transducer

With balanced spatial resolution, penetration depth, and imaging speed, photoacoustic computed tomography (PACT) is promising for clinical translation such as in breast cancer screening, functional brain imaging, and surgical guidance. Typically using a linear ultrasound (US) transducer array, PACT has great flexibility for hand-held applications. However, the linear US transducer array has a limited detection angle range and frequency bandwidth, resulting in limited-view and limited-bandwidth artifacts in the reconstructed PACT images. These artifacts significantly reduce the imaging quality. To address these issues, existing solutions often have to pay the price of system complexity, cost, and/or imaging speed. Here, we propose a deep-learning-based method that explores the Wasserstein generative adversarial network with gradient penalty (WGAN-GP) to reduce the limited-view and limited-bandwidth artifacts in PACT. Compared with existing reconstruction and convolutional neural network approach, our model has shown improvement in imaging quality and resolution. Our results on simulation, phantom, and in vivo data have collectively demonstrated the feasibility of applying WGAN-GP to improve PACT’s image quality without any modification to the current imaging set-up.

Impact statement

This study has the following main impacts. It offers a promising solution for removing limited-view and limited-bandwidth artifact in PACT using a linear-array transducer and conventional image reconstruction, which have long hindered its clinical translation. Our solution shows unprecedented artifact removal ability for in vivo image, which may enable important applications such as imaging tumor angiogenesis and hypoxia. The study reports, for the first time, the use of an advanced deep-learning model based on stabilized generative adversarial network. Our results have demonstrated its superiority over other state-of-the-art deep-learning methods.

Simultaneous Photoacoustic Imaging and Cavitation Mapping in Shockwave Lithotripsy

Kidney stone disease is a major health problem worldwide. Shockwave lithotripsy (SWL), which uses high-energy shockwave pulses to break up kidney stones, is extensively used in clinic. However, despite its noninvasiveness, SWL can produce cavitation in vivo. The rapid expansion and violent collapse of cavitation bubbles in small blood vessels may result in renal vascular injury. To better understand the mechanism of tissue injury and improve treatment safety and efficiency, it is highly desirable to concurrently detect cavitation and vascular injury during SWL. Current imaging modalities used in SWL ( e.g. , C-arm fluoroscopy and B-mode ultrasound) are not sensitive to vascular injuries. By contrast, photoacoustic imaging is a non-invasive and non-radiative imaging modality that is sensitive to blood, by using hemoglobin as the endogenous contrast. Moreover, photoacoustic imaging is also compatible with passive cavitation detection by sharing the ultrasound detection system. Here, we have integrated shockwave treatment, photoacoustic imaging, and passive cavitation detection into a single system. Our experimental results on phantoms and in vivo small animals have collectively demonstrated that the integrated system is capable of capturing shockwave-induced cavitation and the resultant vascular injury simultaneously. We expect that the integrated system, when combined with our recently developed internal-light-illumination photoacoustic imaging, will find important applications for monitoring shockwave-induced vascular injury in deep tissues during SWL.

A Partially-Learned Algorithm for Joint Photo-acoustic Reconstruction and Segmentation

In an inhomogeneously illuminated photoacoustic image, important information like vascular geometry is not readily available, when only the initial pressure is reconstructed. To obtain the desired information, algorithms for image segmentation are often applied as a post-processing step. In this article, we propose to jointly acquire the photoacoustic reconstruction and segmentation, by modifying a recently developed partially learned algorithm based on a convolutional neural network. We investigate the stability of the algorithm against changes in initial pressures and photoacoustic system settings. These insights are used to develop an algorithm that is robust to input and system settings. Our approach can easily be applied to other imaging modalities and can be modified to perform other high-level tasks different from segmentation. The method is validated on challenging synthetic and experimental photoacoustic tomography data in limited angle and limited view scenarios. It is computationally less expensive than classical iterative methods and enables higher quality reconstructions and segmentations than the state-of-the-art learned and non-learned methods.

Image reconstruction based on compressed sensing for sparse-data endoscopic photoacoustic tomography

Endoscopic photoacoustic tomography (EPAT) is an interventional application of photoacoustic tomography (PAT) to visualize anatomical features and functional components of biological cavity structures such as nasal cavity, digestive tract or coronary arterial vessels. One of the main challenges in clinical applicability of EPAT is the incomplete acoustic measurements due to the limited detectors or the limited-view acoustic detection enclosed in the cavity. In this case, conventional image reconstruction methodologies suffer from significantly degraded image quality. This work introduces a compressed-sensing (CS)-based method to reconstruct a high-quality image that represents the initial pressure distribution on a luminal cross-section from incomplete discrete acoustic measurements. The method constructs and trains a complete dictionary for the sparse representation of the photoacoustically-induced acoustic measurements. The sparse representation of the complete acoustic signals is then optimally obtained based on the sparse measurements and a sensing matrix. The complete acoustic signals are recovered from the sparse representation by inverse sparse transformation. The image of the initial pressure distribution is finally reconstructed from the recovered complete signals by using the time reversal (TR) algorithm. It was shown with numerical experiments that high-quality images with reduced under-sampling artifacts can be reconstructed from sparse measurements. The comparison results suggest that the proposed method outperforms the standard TR reconstruction by 40% in terms of the structural similarity of the reconstructed images.

Listening to tissues with new light: recent technological advances in photoacoustic imaging

Photoacoustic tomography (PAT), or optoacoustic tomography, has achieved remarkable progress in the past decade, benefiting from the joint developments in optics, acoustics, chemistry, computing and mathematics. Unlike pure optical or ultrasound imaging, PAT can provide unique optical absorption contrast as well as widely scalable spatial resolution, penetration depth and imaging speed. Moreover, PAT has inherent sensitivity to tissue's functional, molecular, and metabolic state. With these merits, PAT has been applied in a wide range of life science disciplines, and has enabled biomedical research unattainable by other imaging methods. This Review article aims at introducing state-of-the-art PAT technologies and their representative applications. The focus is on recent technological breakthroughs in structural, functional, molecular PAT, including super-resolution imaging, real-time small-animal whole-body imaging, and high-sensitivity functional/molecular imaging. We also discuss the remaining challenges in PAT and envisioned opportunities.

Computationally efficient deep neural network for computed tomography image reconstruction

PURPOSE

Deep neural network-based image reconstruction has demonstrated promising performance in medical imaging for undersampled and low-dose scenarios. However, it requires large amount of memory and extensive time for the training. It is especially challenging to train the reconstruction networks for three-dimensional computed tomography (CT) because of the high resolution of CT images. The purpose of this work is to reduce the memory and time consumption of the training of the reconstruction networks for CT to make it practical for current hardware, while maintaining the quality of the reconstructed images.

METHODS

We unrolled the proximal gradient descent algorithm for iterative image reconstruction to finite iterations and replaced the terms related to the penalty function with trainable convolutional neural networks (CNN). The network was trained greedily iteration by iteration in the image domain on patches, which requires reasonable amount of memory and time on mainstream graphics processing unit (GPU). To overcome the local-minimum problem caused by greedy learning, we used deep UNet as the CNN and incorporated separable quadratic surrogate with ordered subsets for data fidelity, so that the solution could escape from easy local minimums and achieve better image quality.

RESULTS

The proposed method achieved comparable image quality with state-of-the-art neural network for CT image reconstruction on two-dimensional (2D) sparse-view and limited-angle problems on the low-dose CT challenge dataset. The difference in root-mean-square-error (RMSE) and structural similarity index (SSIM) was within [-0.23,0.47] HU and [0,0.001], respectively, with 95% confidence level. For three-dimensional (3D) image reconstruction with ordinary-size CT volume, the proposed method only needed 2 GB graphics processing unit (GPU) memory and 0.45 s per training iteration as minimum requirement, whereas existing methods may require 417 GB and 31 min. The proposed method achieved improved performance compared to total variation- and dictionary learning-based iterative reconstruction for both 2D and 3D problems.

CONCLUSIONS

We proposed a training-time computationally efficient neural network for CT image reconstruction. The proposed method achieved comparable image quality with state-of-the-art neural network for CT reconstruction, with significantly reduced memory and time requirement during training. The proposed method is applicable to 3D image reconstruction problems such as cone-beam CT and tomosynthesis on mainstream GPUs.

Optoacoustic image formation approaches-a clinical perspective

Clinical translation of optoacoustic imaging is fostered by the rapid technical advances in imaging performance as well as the growing number of clinicians recognizing the immense diagnostic potential of this technology. Clinical optoacoustic systems are available in multiple configurations, including hand-held and endoscopic probes as well as raster-scan approaches. The hardware design must be adapted to the accessible portion of the imaged region and other application-specific requirements pertaining the achievable depth, field of view or spatio-temporal resolution. Equally important is the adequate choice of the signal and image processing approach, which is largely responsible for the resulting imaging performance. Thus, new image reconstruction algorithms are constantly evolving in parallel to the newly-developed set-ups. This review focuses on recent progress on optoacoustic image formation algorithms and processing methods in the clinical setting. Major reconstruction challenges include real-time image rendering in two and three dimensions, efficient hybridization with other imaging modalitites as well as accurate interpretation and quantification of bio-markers, herein discussed in the context of ongoing progress in clinical translation.

Networks for Nonlinear Diffusion Problems in Imaging

A multitude of imaging and vision tasks have seen recently a major transformation by deep learning methods and in particular by the application of convolutional neural networks. These methods achieve impressive results, even for applications where it is not apparent that convolutions are suited to capture the underlying physics. In this work, we develop a network architecture based on nonlinear diffusion processes, named DiffNet. By design, we obtain a nonlinear network architecture that is well suited for diffusion-related problems in imaging. Furthermore, the performed updates are explicit, by which we obtain better interpretability and generalisability compared to classical convolutional neural network architectures. The performance of DiffNet is tested on the inverse problem of nonlinear diffusion with the Perona-Malik filter on the STL-10 image dataset. We obtain competitive results to the established U-Net architecture, with a fraction of parameters and necessary training data.

Streak artifact suppression in photoacoustic computed tomography using adaptive back projection

For photoacoustic computed tomography (PACT), an insufficient number of ultrasound detectors can cause serious streak-type artifacts. These artifacts get overlaid on top of image features, and thus locally jeopardize image quality and resolution. Here, a reconstruction algorithm, termed Contamination-Tracing Back-Projection (CTBP), is proposed for the mitigation of streak-type artifacts. During reconstruction, CTBP adaptively adjusts the back-projection weight, whose value is determined by the likelihood of contamination, to minimize the negative influences of strong absorbers. An iterative solution of the eikonal equation is implemented to accurately trace the time of flight of different pixels. Numerical, phantom and in vivo experiments demonstrate that CTBP can dramatically suppress streak artifacts in PACT and improve image quality.

Beltrami-net: domain-independent deep D-bar learning for absolute imaging with electrical impedance tomography (a-EIT)

OBJECTIVE

To develop, and demonstrate the feasibility of, a novel image reconstruction method for absolute electrical impedance tomography (a-EIT) that pairs deep learning techniques with real-time robust D-bar methods and examine the influence of prior information on the reconstruction.

APPROACH

A D-bar method is paired with a trained convolutional neural network (CNN) as a post-processing step. Training data is simulated for the network using no knowledge of the boundary shape by using an associated nonphysical Beltrami equation rather than simulating the traditional current and voltage data specific to a given domain. This allows the training data to be boundary shape independent. The method is tested on experimental data from two EIT systems (ACT4 and KIT4) with separate training sets of varying prior information.

MAIN RESULTS

Post-processing the D-bar images with a CNN produces significant improvements in image quality measured by structural SIMilarity indices (SSIMs) as well as relative [Formula: see text] and [Formula: see text] image errors.

SIGNIFICANCE

This work demonstrates that more general networks can be trained without being specific about boundary shape, a key challenge in EIT image reconstruction. The work is promising for future studies involving databases of anatomical atlases.

Accelerated Correction of Reflection Artifacts by Deep Neural Networks in Photo-Acoustic Tomography

Photo-Acoustic Tomography (PAT) is an emerging non-invasive hybrid modality driven by a constant yearning for superior imaging performance. The image quality, however, hinges on the acoustic reflection, which may compromise the diagnostic performance. To address this challenge, we propose to incorporate a deep neural network into conventional iterative algorithms to accelerate and improve the correction of reflection artifacts. Based on the simulated PAT dataset from computed tomography (CT) scans, this network-accelerated reconstruction approach is shown to outperform two state-of-the-art iterative algorithms in terms of the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) in the presence of noise. The proposed network also demonstrates considerably higher computational efficiency than conventional iterative algorithms, which are time-consuming and cumbersome.

[Review on photoacoustic tomographic image reconstruction for acoustically heterogeneous tissues]

Acoustic properties of biological tissues usually vary inhomogeneously in space. Tissues with different chemical composition often have different acoustic properties. The assumption of acoustic homogeneity may lead to blurred details, misalignment of targets and artifacts in the reconstructed photoacoustic tomography (PAT) images. This paper summarizes the main solutions to PAT imaging of acoustically heterogeneous tissues, including the variable sound speed and acoustic attenuation. The advantages and limits of the methods are discussed and the possible future development is prospected.

Fully Dense UNet for 2-D Sparse Photoacoustic Tomography Artifact Removal

Photoacoustic imaging is an emerging imaging modality that is based upon the photoacoustic effect. In photoacoustic tomography (PAT), the induced acoustic pressure waves are measured by an array of detectors and used to reconstruct an image of the initial pressure distribution. A common challenge faced in PAT is that the measured acoustic waves can only be sparsely sampled. Reconstructing sparsely sampled data using standard methods results in severe artifacts that obscure information within the image. We propose a modified convolutional neural network (CNN) architecture termed fully dense UNet (FD-UNet) for removing artifacts from two-dimensional PAT images reconstructed from sparse data and compare the proposed CNN with the standard UNet in terms of reconstructed image quality.

Dictionary learning sparse-sampling reconstruction method for in-vivo 3D photoacoustic computed tomography

The sparse transforms currently used in the model-based reconstruction method for photoacoustic computed tomography (PACT) are predefined and they typically cannot capture the underlying features of the specific data sets adequately, thus limiting the high-quality recovery of photoacoustic images. In this work, we present an advanced reconstruction model using the K-VSD dictionary learning technique and present the in vivo results after adapting the model into the 3D PACT system. The in vivo experiments were performed on an IRB approved human hand and two rats. When compared to the traditional sparse transform, experimental results using our proposed method improved accuracy and contrast to noise ration of the reconstructed photoacoustic images, on average, by 3.7 and 1.8 times in the case of 50% sparse-sampling rate, respectively. We also compared the performance of our algorithm against other techniques, and imaging speed was 60% faster than other approaches. Our system would require sparse-transducer array and lower number of data acquisition hardware (DAQs) potentially reducing the cost of the system. Thus, our work provides a new way for reconstructing photoacoustic images, and it would enable the development of new high-speed low-cost 3D PACT for various biomedical applications.

Confidence Estimation for Machine Learning-Based Quantitative Photoacoustics

In medical applications, the accuracy and robustness of imaging methods are of crucial importance to ensure optimal patient care. While photoacoustic imaging (PAI) is an emerging modality with promising clinical applicability, state-of-the-art approaches to quantitative photoacoustic imaging (qPAI), which aim to solve the ill-posed inverse problem of recovering optical absorption from the measurements obtained, currently cannot comply with these high standards. This can be attributed to the fact that existing methods often rely on several simplifying a priori assumptions of the underlying physical tissue properties or cannot deal with realistic noise levels. In this manuscript, we address this issue with a new method for estimating an indicator of the uncertainty of an estimated optical property. Specifically, our method uses a deep learning model to compute error estimates for optical parameter estimations of a qPAI algorithm. Functional tissue parameters, such as blood oxygen saturation, are usually derived by averaging over entire signal intensity-based regions of interest (ROIs). Therefore, we propose to reduce the systematic error of the ROI samples by additionally discarding those pixels for which our method estimates a high error and thus a low confidence. In silico experiments show an improvement in the accuracy of optical absorption quantification when applying our method to refine the ROI, and it might thus become a valuable tool for increasing the robustness of qPAI methods.

Photoacoustic imaging reconstruction using combined nonlocal patch and total-variation regularization for straight-line scanning

Background

For practical straight-line scanning in photoacoustic imaging (PAI), serious artifacts caused by missing data will occur. Traditional total variation (TV)-based algorithms fail to obtain satisfactory results, with an over-smoothed and blurred geometric structure. Therefore, it is important to develop a new algorithm to improve the quality of practical straight-line reconstructed images.

Methods

In this paper, a combined nonlocal patch and TV-based regularization model for PAI reconstruction is proposed to solve these problems. A modified adaptive nonlocal weight function is adopted to provide more reliable estimations for the similarities between patches. Similar patches are searched for throughout the entire image; thus, this model realizes adaptive search for the neighborhood of the patch. The optimization problem is simplified to a common iterative PAI reconstruction problem.

Results and conclusion

The proposed algorithm is validated by a series of numerical simulations and an in vitro experiment for straight-line scanning. The results of patch-TV are compared to those of two mainstream TV-based algorithms as well as the iterative algorithm only with patch-based regularization. Moreover, the peak signal-to-noise ratio, the noise robustness, and the convergence and calculation speeds are compared and discussed. The results show that the proposed patch-TV yields significant improvement over the other three algorithms qualitatively and quantitatively. These simulations and experiment indicate that the patch-TV algorithm successfully solves the problems of PAI reconstruction and is highly effective in practical PAI applications.

Photoacoustic tomography of blood oxygenation: A mini review

Photoacoustic tomography (PAT) is a hybrid imaging modality that combines rich contrast of optical excitation and deep penetration of ultrasound detection. With its unique optical absorption contrast mechanism, PAT is inherently sensitive to the functional and molecular information of biological tissues, and thus has been widely used in preclinical and clinical studies. Among many functional capabilities of PAT, measuring blood oxygenation is arguably one of the most important applications, and has been widely performed in photoacoustic studies of brain functions, tumor hypoxia, wound healing, and cancer therapy. Yet, the complex optical conditions of biological tissues, especially the strong wavelength-dependent optical attenuation, have long hurdled the PAT measurement of blood oxygenation at depths beyond a few millimeters. A variety of PAT methods have been developed to improve the accuracy of blood oxygenation measurement, using novel laser illumination schemes, oxygen-sensitive fluorescent dyes, comprehensive mathematic models, or prior information provided by complementary imaging modalities. These novel methods have made exciting progress, while several challenges remain. This concise review aims to introduce the recent developments in photoacoustic blood oxygenation measurement, compare each method's advantages and limitations, highlight their representative applications, and discuss the remaining challenges for future advances.

Context encoding enables machine learning-based quantitative photoacoustics

Real-time monitoring of functional tissue parameters, such as local blood oxygenation, based on optical imaging could provide groundbreaking advances in the diagnosis and interventional therapy of various diseases. Although photoacoustic (PA) imaging is a modality with great potential to measure optical absorption deep inside tissue, quantification of the measurements remains a major challenge. We introduce the first machine learning-based approach to quantitative PA imaging (qPAI), which relies on learning the fluence in a voxel to deduce the corresponding optical absorption. The method encodes relevant information of the measured signal and the characteristics of the imaging system in voxel-based feature vectors, which allow the generation of thousands of training samples from a single simulated PA image. Comprehensive in silico experiments suggest that context encoding-qPAI enables highly accurate and robust quantification of the local fluence and thereby the optical absorption from PA images.

All-optical forward-viewing photoacoustic probe for high-resolution 3D endoscopy

A miniature forward-viewing endoscopic probe that provides high-resolution 3D photoacoustic images is demonstrated. The probe is of outer diameter 3.2 mm and comprised of a transparent Fabry-Pérot (FP) polymer-film ultrasound sensor that is located at the distal end of a rigid optical fiber bundle. Excitation laser pulses are coupled simultaneously into all cores of the bundle and are transmitted through the FP sensor to provide wide-field tissue illumination at the distal end. The resulting photoacoustic waves are mapped in 2D by sequentially scanning the input end of the bundle with an interrogation laser beam in order to individually address different points on the FP sensor. In this way, the sensor acts as a high-density ultrasound array that is comprised of 50,000 individual elements, each of which is 12 µm in diameter, within the 3.2 mm diameter footprint of the probe. The fine spatial sampling that this affords, along with the wide bandwidth (f _-3dB =34 MHz) of the sensor, enables a high-resolution photoacoustic image to be reconstructed. The measured on-axis lateral resolution of the probe was depth-dependent and ranged from 45-170 µm for depths between 1 and 7 mm, and the vertical resolution was 31 µm over the same depth range. The system was evaluated by acquiring 3D images of absorbing phantoms and the microvascular anatomies of a duck embryo and mouse skin. Excellent image fidelity was demonstrated. It is anticipated that this type of probe could find application as a tool for guiding laparoscopic procedures, fetal surgery and other minimally invasive interventions that require a millimeter-scale forward-viewing 3D photoacoustic imaging probe.