Deep Learning for Automatic Segmentation of Hybrid Optoacoustic Ultrasound (OPUS) Images

Recurrent and convolutional neural networks for sequential multispectral optoacoustic tomography (MSOT) imaging

Multispectral optoacoustic tomography (MSOT) is a beneficial technique for diagnosing and analyzing biological samples since it provides meticulous details in anatomy and physiology. However, acquiring high through-plane resolution volumetric MSOT is time-consuming. Here, we propose a deep learning model based on hybrid recurrent and convolutional neural networks to generate sequential cross-sectional images for an MSOT system. This system provides three modalities (MSOT, ultrasound, and optoacoustic imaging of a specific exogenous contrast agent) in a single scan. This study used ICG-conjugated nanoworms particles (NWs-ICG) as the contrast agent. Instead of acquiring seven images with a step size of 0.1 mm, we can receive two images with a step size of 0.6 mm as input for the proposed deep learning model. The deep learning model can generate five other images with a step size of 0.1 mm between these two input images meaning we can reduce acquisition time by approximately 71%.

Application of deep neural networks for inferring pressure in polymeric acoustic transponders/sensors

Passive sensor-transponders have raised interest for the last few decades, due to their capability of low-cost remote monitoring without the need for energy storage. Their operating principle includes receiving a signal from a source and then reflecting the signal. While well-established transponders operate through electromagnetic antennas, those with a fully acoustic design have advantages such as lower cost and simplicity. Therefore, detection of pressures using the ultrasound signal that is backscattered from an acoustic resonator has been of interest recently. In order to infer the pressure from the backscattered signal, the established approach has been based upon the principle of detection of the shift to the frequency of resonance. Nevertheless, regression of the pressure from the signal with a small error is challenging and has been subject to research. Here in this paper, we explore an approach that employs deep learning for inferring pressure from the ultrasound reflections of polymeric resonators. We assess if neural network regressors can efficiently infer pressure reflected from a fully acoustic transponder. For this purpose, we compare the performance of several regressors such as a convolutional neural network, a network inspired by the ResNet, and a fully connected neural network. We observe that deep neural networks are advantageous in inferring pressure information with a minimal need for analyzing the signal. Our work suggests that a deep learning approach has the potential to be integrated with or replace other traditional approaches for inferring pressure from an ultrasound signal reflected from fully acoustic transponders or passive sensors.

Segmentation of Arm Ultrasound Images in Breast Cancer-Related Lymphedema: A Database and Deep Learning Algorithm

OBJECTIVE

Breast cancer treatment often causes the removal of or damage to lymph nodes of the patient's lymphatic drainage system. This side effect is the origin of Breast Cancer-Related Lymphedema (BCRL), referring to a noticeable increase in excess arm volume. Ultrasound imaging is a preferred modality for the diagnosis and progression monitoring of BCRL because of its low cost, safety, and portability. As the affected and unaffected arms look similar in B-mode ultrasound images, the thickness of the skin, subcutaneous fat, and muscle have been shown to be important biomarkers for this task. The segmentation masks are also helpful in monitoring the longitudinal changes in morphology and mechanical properties of tissue layers.

METHODS

For the first time, a publicly available ultrasound dataset containing the Radio-Frequency (RF) data of 39 subjects and manual segmentation masks by two experts, are provided. Inter- and intra-observer reproducibility studies performed on the segmentation maps show a high Dice Score Coefficient (DSC) of 0.94±0.08 and 0.92±0.06, respectively. Gated Shape Convolutional Neural Network (GSCNN) is modified for precise automatic segmentation of tissue layers, and its generalization performance is improved by the CutMix augmentation strategy.

RESULTS

We got an average DSC of 0.87±0.11 on the test set, which confirms the high performance of the method.

CONCLUSION

Automatic segmentation can pave the way for convenient and accessible staging of BCRL, and our dataset can facilitate development and validation of those methods.

SIGNIFICANCE

Timely diagnosis and treatment of BCRL have crucial importance in preventing irreversible damage.

Artificial Intelligence framework with traditional computer vision and deep learning approaches for optimal automatic segmentation of left ventricle with scar

Automatic segmentation of the cardiac left ventricle with scars remains a challenging and clinically significant task, as it is essential for patient diagnosis and treatment pathways. This study aimed to develop a novel framework and cost function to achieve optimal automatic segmentation of the left ventricle with scars using LGE-MRI images. To ensure the generalization of the framework, an unbiased validation protocol was established using out-of-distribution (OOD) internal and external validation cohorts, and intra-observation and inter-observer variability ground truths. The framework employs a combination of traditional computer vision techniques and deep learning, to achieve optimal segmentation results. The traditional approach uses multi-atlas techniques, active contours, and k-means methods, while the deep learning approach utilizes various deep learning techniques and networks. The study found that the traditional computer vision technique delivered more accurate results than deep learning, except in cases where there was breath misalignment error. The optimal solution of the framework achieved robust and generalized results with Dice scores of 82.8 ± 6.4% and 72.1 ± 4.6% in the internal and external OOD cohorts, respectively. The developed framework offers a high-performance solution for automatic segmentation of the left ventricle with scars using LGE-MRI. Unlike existing state-of-the-art approaches, it achieves unbiased results across different hospitals and vendors without the need for training or tuning in hospital cohorts. This framework offers a valuable tool for experts to accomplish the task of fully automatic segmentation of the left ventricle with scars based on a single-modality cardiac scan.

Applications of Deep Learning Algorithms to Ultrasound Imaging Analysis in Preclinical Studies on In Vivo Animals

BACKGROUND AND AIM

Ultrasound (US) imaging is increasingly preferred over other more invasive modalities in preclinical studies using animal models. However, this technique has some limitations, mainly related to operator dependence. To overcome some of the current drawbacks, sophisticated data processing models are proposed, in particular artificial intelligence models based on deep learning (DL) networks. This systematic review aims to overview the application of DL algorithms in assisting US analysis of images acquired in in vivo preclinical studies on animal models.

METHODS

A literature search was conducted using the Scopus and PubMed databases. Studies published from January 2012 to November 2022 that developed DL models on US images acquired in preclinical/animal experimental scenarios were eligible for inclusion. This review was conducted according to PRISMA guidelines.

RESULTS

Fifty-six studies were enrolled and classified into five groups based on the anatomical district in which the DL models were used. Sixteen studies focused on the cardiovascular system and fourteen on the abdominal organs. Five studies applied DL networks to images of the musculoskeletal system and eight investigations involved the brain. Thirteen papers, grouped under a miscellaneous category, proposed heterogeneous applications adopting DL systems. Our analysis also highlighted that murine models were the most common animals used in in vivo studies applying DL to US imaging.

CONCLUSION

DL techniques show great potential in terms of US images acquired in preclinical studies using animal models. However, in this scenario, these techniques are still in their early stages, and there is room for improvement, such as sample sizes, data preprocessing, and model interpretability.

Automatic detection of prostate cancer grades and chronic prostatitis in biparametric MRI

BACKGROUND AND OBJECTIVE

With emerging evidence to improve prostate cancer (PCa) screening, multiparametric magnetic prostate imaging is becoming an essential noninvasive component of the diagnostic routine. Computer-aided diagnostic (CAD) tools powered by deep learning can help radiologists interpret multiple volumetric images. In this work, our objective was to examine promising methods recently proposed in the multigrade prostate cancer detection task and to suggest practical considerations regarding model training in this context.

METHODS

We collected 1647 fine-grained biopsy-confirmed findings, including Gleason scores and prostatitis, to form a training dataset. In our experimental framework for lesion detection, all models utilized 3D nnU-Net architecture that accounts for anisotropy in the MRI data. First, we explore an optimal range of b-values for diffusion-weighted imaging (DWI) modality and its effect on the detection of clinically significant prostate cancer (csPCa) and prostatitis using deep learning, as the optimal range is not yet clearly defined in this domain. Next, we propose a simulated multimodal shift as a data augmentation technique to compensate for the multimodal shift present in the data. Third, we study the effect of incorporating the prostatitis class alongside cancer-related findings at three different granularities of the prostate cancer class (coarse, medium, and fine) and its impact on the detection rate of the target csPCa. Furthermore, ordinal and one-hot encoded (OHE) output formulations were tested.

RESULTS

An optimal model configuration with fine class granularity (prostatitis included) and OHE has scored the lesion-wise partial Free-Response Receiver Operating Characteristic (FROC) area under the curve (AUC) of 1.94 (CI 95%: 1.76-2.11) and patient-wise ROC AUC of 0.874 (CI 95%: 0.793-0.938) in the detection of csPCa. Inclusion of the auxiliary prostatitis class has demonstrated a stable relative improvement in specificity at a false positive rate (FPR) of 1.0 per patient, with an increase of 3%, 7%, and 4% for coarse, medium, and fine class granularities.

CONCLUSIONS

This paper examines several configurations for model training in the biparametric MRI setup and proposes optimal value ranges. It also shows that the fine-grained class configuration, including prostatitis, is beneficial for detecting csPCa. The ability to detect prostatitis in all low-risk cancer lesions suggests the potential to improve the quality of the early diagnosis of prostate diseases. It also implies an improved interpretability of the results by the radiologist.

Monitoring mouse brain perfusion with hybrid magnetic resonance optoacoustic tomography

Progress in brain research critically depends on the development of next-generation multi-modal imaging tools capable of capturing transient functional events and multiplexed contrasts noninvasively and concurrently, thus enabling a holistic view of dynamic events in vivo. Here we report on a hybrid magnetic resonance and optoacoustic tomography (MROT) system for murine brain imaging, which incorporates an MR-compatible spherical matrix array transducer and fiber-based light illumination into a 9.4 T small animal scanner. An optimized radiofrequency coil has further been devised for whole-brain interrogation. System's utility is showcased by acquiring complementary angiographic and soft tissue anatomical contrast along with simultaneous dual-modality visualization of contrast agent dynamics in vivo.

A Signal Domain Object Segmentation Method for Ultrasound and Photoacoustic Computed Tomography

Image segmentation is important in improving the diagnostic capability of ultrasound computed tomography (USCT) and photoacoustic computed tomography (PACT), as it can be included in the image reconstruction process to improve image quality and quantification abilities. Segmenting the imaged object out of the background using image domain methods is easily complicated by low contrast, noise, and artifacts in the reconstructed image. Here, we introduce a new signal domain object segmentation method for USCT and PACT which does not require image reconstruction beforehand and is automatic, robust, computationally efficient, accurate, and straightforward. We first establish the relationship between the time-of-flight of the received first arrival waves and the object's boundary which is described by ellipse equations. Then, we show that the ellipses are tangent to the boundary. By looking for tangent points on the common tangent of neighboring ellipses, the boundary can be approximated with high fidelity. Imaging experiments of human fingers and mice cross-sections showed that our method provided equivalent or better segmentations than the optimal ones by active contours. In summary, our method greatly reduces the overall complexity of object segmentation and shows great potential in eliminating user dependency without sacrificing segmentation accuracy. The method can be further seamlessly incorporated into algorithms for other processing purposes in USCT and PACT, such as high-quality image reconstruction.

Deep learning facilitates fully automated brain image registration of optoacoustic tomography and magnetic resonance imaging

Multispectral optoacoustic tomography (MSOT) is an emerging optical imaging method providing multiplex molecular and functional information from the rodent brain. It can be greatly augmented by magnetic resonance imaging (MRI) which offers excellent soft-tissue contrast and high-resolution brain anatomy. Nevertheless, registration of MSOT-MRI images remains challenging, chiefly due to the entirely different image contrast rendered by these two modalities. Previously reported registration algorithms mostly relied on manual user-dependent brain segmentation, which compromised data interpretation and quantification. Here we propose a fully automated registration method for MSOT-MRI multimodal imaging empowered by deep learning. The automated workflow includes neural network-based image segmentation to generate suitable masks, which are subsequently registered using an additional neural network. The performance of the algorithm is showcased with datasets acquired by cross-sectional MSOT and high-field MRI preclinical scanners. The automated registration method is further validated with manual and half-automated registration, demonstrating its robustness and accuracy.

Semantic segmentation of multispectral photoacoustic images using deep learning

Photoacoustic (PA) imaging has the potential to revolutionize functional medical imaging in healthcare due to the valuable information on tissue physiology contained in multispectral photoacoustic measurements. Clinical translation of the technology requires conversion of the high-dimensional acquired data into clinically relevant and interpretable information. In this work, we present a deep learning-based approach to semantic segmentation of multispectral photoacoustic images to facilitate image interpretability. Manually annotated photoacoustic and ultrasound imaging data are used as reference and enable the training of a deep learning-based segmentation algorithm in a supervised manner. Based on a validation study with experimentally acquired data from 16 healthy human volunteers, we show that automatic tissue segmentation can be used to create powerful analyses and visualizations of multispectral photoacoustic images. Due to the intuitive representation of high-dimensional information, such a preprocessing algorithm could be a valuable means to facilitate the clinical translation of photoacoustic imaging.

Deep-Learning-Based Algorithm for the Removal of Electromagnetic Interference Noise in Photoacoustic Endoscopic Image Processing

Despite all the expectations for photoacoustic endoscopy (PAE), there are still several technical issues that must be resolved before the technique can be successfully translated into clinics. Among these, electromagnetic interference (EMI) noise, in addition to the limited signal-to-noise ratio (SNR), have hindered the rapid development of related technologies. Unlike endoscopic ultrasound, in which the SNR can be increased by simply applying a higher pulsing voltage, there is a fundamental limitation in leveraging the SNR of PAE signals because they are mostly determined by the optical pulse energy applied, which must be within the safety limits. Moreover, a typical PAE hardware situation requires a wide separation between the ultrasonic sensor and the amplifier, meaning that it is not easy to build an ideal PAE system that would be unaffected by EMI noise. With the intention of expediting the progress of related research, in this study, we investigated the feasibility of deep-learning-based EMI noise removal involved in PAE image processing. In particular, we selected four fully convolutional neural network architectures, U-Net, Segnet, FCN-16s, and FCN-8s, and observed that a modified U-Net architecture outperformed the other architectures in the EMI noise removal. Classical filter methods were also compared to confirm the superiority of the deep-learning-based approach. Still, it was by the U-Net architecture that we were able to successfully produce a denoised 3D vasculature map that could even depict the mesh-like capillary networks distributed in the wall of a rat colorectum. As the development of a low-cost laser diode or LED-based photoacoustic tomography (PAT) system is now emerging as one of the important topics in PAT, we expect that the presented AI strategy for the removal of EMI noise could be broadly applicable to many areas of PAT, in which the ability to apply a hardware-based prevention method is limited and thus EMI noise appears more prominently due to poor SNR.

Recent Technical Advances in Accelerating the Clinical Translation of Small Animal Brain Imaging: Hybrid Imaging, Deep Learning, and Transcriptomics

Small animal models play a fundamental role in brain research by deepening the understanding of the physiological functions and mechanisms underlying brain disorders and are thus essential in the development of therapeutic and diagnostic imaging tracers targeting the central nervous system. Advances in structural, functional, and molecular imaging using MRI, PET, fluorescence imaging, and optoacoustic imaging have enabled the interrogation of the rodent brain across a large temporal and spatial resolution scale in a non-invasively manner. However, there are still several major gaps in translating from preclinical brain imaging to the clinical setting. The hindering factors include the following: (1) intrinsic differences between biological species regarding brain size, cell type, protein expression level, and metabolism level and (2) imaging technical barriers regarding the interpretation of image contrast and limited spatiotemporal resolution. To mitigate these factors, single-cell transcriptomics and measures to identify the cellular source of PET tracers have been developed. Meanwhile, hybrid imaging techniques that provide highly complementary anatomical and molecular information are emerging. Furthermore, deep learning-based image analysis has been developed to enhance the quantification and optimization of the imaging protocol. In this mini-review, we summarize the recent developments in small animal neuroimaging toward improved translational power, with a focus on technical improvement including hybrid imaging, data processing, transcriptomics, awake animal imaging, and on-chip pharmacokinetics. We also discuss outstanding challenges in standardization and considerations toward increasing translational power and propose future outlooks.

Full-view in vivo skin and blood vessels profile segmentation in photoacoustic imaging based on deep learning

Photoacoustic (PA) microscopy allows imaging of the soft biological tissue based on optical absorption contrast and spatial ultrasound resolution. One of the major applications of PA imaging is its characterization of microvasculature. However, the strong PA signal from skin layer overshadowed the subcutaneous blood vessels leading to indirectly reconstruct the PA images in human study. Addressing the present situation, we examined a deep learning (DL) automatic algorithm to achieve high-resolution and high-contrast segmentation for widening PA imaging applications. In this research, we propose a DL model based on modified U-Net for extracting the relationship features between amplitudes of the generated PA signal from skin and underlying vessels. This study illustrates the broader potential of hybrid complex network as an automatic segmentation tool for the in vivo PA imaging. With DL-infused solution, our result outperforms the previous studies with achieved real-time semantic segmentation on large-size high-resolution PA images.

Random Fourier Features-Based Deep Learning Improvement with Class Activation Interpretability for Nerve Structure Segmentation

Peripheral nerve blocking (PNB) is a standard procedure to support regional anesthesia. Still, correct localization of the nerve's structure is needed to avoid adverse effects; thereby, ultrasound images are used as an aid approach. In addition, image-based automatic nerve segmentation from deep learning methods has been proposed to mitigate attenuation and speckle noise ultrasonography issues. Notwithstanding, complex architectures highlight the region of interest lacking suitable data interpretability concerning the learned features from raw instances. Here, a kernel-based deep learning enhancement is introduced for nerve structure segmentation. In a nutshell, a random Fourier features-based approach was utilized to complement three well-known semantic segmentation architectures, e.g., fully convolutional network, U-net, and ResUnet. Moreover, two ultrasound image datasets for PNB were tested. Obtained results show that our kernel-based approach provides a better generalization capability from image segmentation-based assessments on different nerve structures. Further, for data interpretability, a semantic segmentation extension of the GradCam++ for class-activation mapping was used to reveal relevant learned features separating between nerve and background. Thus, our proposal favors both straightforward (shallow) and complex architectures (deeper neural networks).

Binary photoacoustic tomography for improved vasculature imaging

SIGNIFICANCE

The proposed binary tomography approach was able to recover the vasculature structures accurately, which could potentially enable the utilization of binary tomography algorithm in scenarios such as therapy monitoring and hemorrhage detection in different organs.

AIM

Photoacoustic tomography (PAT) involves reconstruction of vascular networks having direct implications in cancer research, cardiovascular studies, and neuroimaging. Various methods have been proposed for recovering vascular networks in photoacoustic imaging; however, most methods are two-step (image reconstruction and image segmentation) in nature. We propose a binary PAT approach wherein direct reconstruction of vascular network from the acquired photoacoustic sinogram data is plausible.

APPROACH

Binary tomography approach relies on solving a dual-optimization problem to reconstruct images with every pixel resulting in a binary outcome (i.e., either background or the absorber). Further, the binary tomography approach was compared against backprojection, Tikhonov regularization, and sparse recovery-based schemes.

RESULTS

Numerical simulations, physical phantom experiment, and in-vivo rat brain vasculature data were used to compare the performance of different algorithms. The results indicate that the binary tomography approach improved the vasculature recovery by 10% using in-silico data with respect to the Dice similarity coefficient against the other reconstruction methods.

CONCLUSION

The proposed algorithm demonstrates superior vasculature recovery with limited data both visually and based on quantitative image metrics.

A Deep Learning-Based Automatic First-Arrival Picking Method for Ultrasound Sound-Speed Tomography

Ultrasound sound-speed tomography (USST) has shown great prospects for breast cancer diagnosis due to its advantages of nonradiation, low cost, 3-D breast images, and quantitative indicators. However, the reconstruction quality of USST is highly dependent on the first-arrival picking of the transmission wave. Traditional first-arrival picking methods have low accuracy and noise robustness. To improve the accuracy and robustness, we introduced a self-attention mechanism into the bidirectional long short-term memory (BLSTM) network and proposed the self-attention BLSTM (SAT-BLSTM) network. The proposed method predicts the probability of the first-arrival time and selects the time with maximum probability. A numerical simulation and prototype experiment were conducted. In the numerical simulation, the proposed SAT-BLSTM showed the best results. For signal-to-noise ratios (SNRs) of 50, 30, and 15 dB, the mean absolute errors (MAEs) were 48, 49, and 76 ns, respectively. The BLSTM had the second-best results, with MAEs of 55, 56, and 85 ns, respectively. The MAEs of the Akaike information criterion (AIC) method were 57, 296, and 489 ns, respectively. In the prototype experiment, the MAEs of the SAT-BLSTM, the BLSTM, and the AIC were 94, 111, and 410 ns, respectively.

Deep learning of image- and time-domain data enhances the visibility of structures in optoacoustic tomography

Images rendered with common optoacoustic system implementations are often afflicted with distortions and poor visibility of structures, hindering reliable image interpretation and quantification of bio-chrome distribution. Among the practical limitations contributing to artifactual reconstructions are insufficient tomographic detection coverage and suboptimal illumination geometry, as well as inability to accurately account for acoustic reflections and speed of sound heterogeneities in the imaged tissues. Here we developed a convolutional neural network (CNN) approach for enhancement of optoacoustic image quality which combines training on both time-resolved signals and tomographic reconstructions. Reference human finger data for training the CNN were recorded using a full-ring array system that provides optimal tomographic coverage around the imaged object. The reconstructions were further refined with a dedicated algorithm that minimizes acoustic reflection artifacts induced by acoustically mismatch structures, such as bones. The combined methodology is shown to outperform other learning-based methods solely operating on image-domain data.

Multi-scale optoacoustic molecular imaging of brain diseases

The ability to non-invasively visualize endogenous chromophores and exogenous probes and sensors across the entire rodent brain with the high spatial and temporal resolution has empowered optoacoustic imaging modalities with unprecedented capacities for interrogating the brain under physiological and diseased conditions. This has rapidly transformed optoacoustic microscopy (OAM) and multi-spectral optoacoustic tomography (MSOT) into emerging research tools to study animal models of brain diseases. In this review, we describe the principles of optoacoustic imaging and showcase recent technical advances that enable high-resolution real-time brain observations in preclinical models. In addition, advanced molecular probe designs allow for efficient visualization of pathophysiological processes playing a central role in a variety of neurodegenerative diseases, brain tumors, and stroke. We describe outstanding challenges in optoacoustic imaging methodologies and propose a future outlook.