Multi-Center MRI Carotid Plaque Component Segmentation Using Feature Normalization and Transfer Learning

Validating Joint Acoustic Emissions Models as a Generalizable Predictor of Joint Health

Joint acoustic emissions (JAEs) have been used as a non-invasive sensing modality of joint health for different conditions such as acute injuries, osteoarthritis (OA), and rheumatoid arthritis (RA). Recent hardware improvements for sensing JAEs have made at-home sensing to supplement clinical visits a possibility. To complement these advances, models must be improved for JAEs to function as generalizable predictors of joint health. Addressing this need, this work investigates the effects of recording setup, location-specific factors, and participant population on previously validated JAE models. The effect of recording setup is first investigated by testing a model developed previously for a wearable brace to predict erythrocyte sedimentation rate (ESR) in participants with RA on benchtop data, resulting in an area under the receiver-operating characteristic curve (AUC), sensitivity, and specificity of 0.79, 0.73, and 0.81 respectively. Investigating the effects of participant population type and location-specific factors, a feature-based model and a convolutional neural network (CNN) were both trained with healthy and RA data to predict ESR level, and then tested on a new dataset containing healthy, pre-radiographic osteoarthritis (Pre-OA), and OA data. The feature-based model had an AUC of 0.69 and 0.94, a sensitivity of 0.38 and 0.80, and a sensitivity of 1, while the CNN had an AUC of 0.85 and 0.99, a sensitivity of 0.50 and 1, and a specificity of 0.90 for detecting Pre-OA and OA respectively. The ability to generalize models across setup, location, and participant population provides a foundation for using JAEs as a measure of joint health.

Nested star-shaped objects segmentation using diameter annotations

Most current deep learning based approaches for image segmentation require annotations of large datasets, which limits their application in clinical practice. We observe a mismatch between the voxelwise ground-truth that is required to optimize an objective at a voxel level and the commonly used, less time-consuming clinical annotations seeking to characterize the most important information about the patient (diameters, counts, etc.). In this study, we propose to bridge this gap for the case of multiple nested star-shaped objects (e.g., a blood vessel lumen and its outer wall) by optimizing a deep learning model based on diameter annotations. This is achieved by extracting in a differentiable manner the boundary points of the objects at training time, and by using this extraction during the backpropagation. We evaluate the proposed approach on segmentation of the carotid artery lumen and wall from multisequence MR images, thus reducing the annotation burden to only four annotated landmarks required to measure the diameters in the direction of the vessel's maximum narrowing. Our experiments show that training based on diameter annotations produces state-of-the-art weakly supervised segmentations and performs reasonably compared to full supervision. We made our code publicly available at https://gitlab.com/radiology/aim/carotid-artery-image-analysis/nested-star-shaped-objects.

BP-Net: Boundary and perfusion feature guided dual-modality ultrasound video analysis network for fibrous cap integrity assessment

Ultrasonography is one of the main imaging methods for monitoring and diagnosing atherosclerosis due to its non-invasiveness and low-cost. Automatic differentiation of carotid plaque fibrous cap integrity by using multi-modal ultrasound videos has significant diagnostic and prognostic value for cardiovascular and cerebrovascular disease patients. However, the task faces several challenges, including high variation in plaque location and shape, the absence of analysis mechanism focusing on fibrous cap, the lack of effective mechanism to capture the relevance among multi-modal data for feature fusion and selection, etc. To overcome these challenges, we propose a new target boundary and perfusion feature guided video analysis network (BP-Net) based on conventional B-mode ultrasound and contrast-enhanced ultrasound videos for assessing the integrity of fibrous cap. Based on our previously proposed plaque auto-tracking network, in our BP-Net, we further introduce the plaque edge attention module and reverse mechanism to focus the dual video analysis on the fiber cap of plaques. Moreover, to fully explore the rich information on the fibrous cap and inside/outside of the plaque, we propose a feature fusion module for B-mode and contrast video to filter out the most valuable features for fibrous cap integrity assessment. Finally, multi-head convolution attention is proposed and embedded into transformer-based network, which captures semantic features and global context information to obtain accurate evaluation of fibrous caps integrity. The experimental results demonstrate that the proposed method has high accuracy and generalizability with an accuracy of 92.35% and an AUC of 0.935, which outperforms than the state-of-the-art deep learning based methods. A series of comprehensive ablation studies suggest the effectiveness of each proposed component and show great potential in clinical application.

CPTV: Classification by tracking of carotid plaque in ultrasound videos

The risk assessment of carotid plaque is strongly related to the plaque echo status in ultrasound. However, the echo classification of carotid plaques based on ultrasound remains challenging due to the changes in plaque shape and semantics, along with the complex vascular environment. This study proposed a framework for Classification of Plaque by Tracking Videos (CPTV). To the best of our knowledge, this is the first study on plaque classification by tracking ultrasound video rather than a sonographic view, which achieves accurate localization and stable echo classification. In the tracking task, Multi-scale Decoupling Tracking (MDTrack) module including Multi-scale Dilated Encoder (MDE) and Internal-Exterior Feature Decoupling (IEFD) was proposed to solve the problems caused by shape and semantic variations to achieve accurate plaque localization in ultrasound. In the classification task, the Tracking-assisted 3D Attention (T3D-Attention) module included recombination and 3D-Attention extracted plaque features and echo-related features in the vascular environment. The experiments demonstrated that the performance of CPTV is better than current mainstream tracking and classification methods, indicating that the tracking-assistance classification is a kind of enhancement method with high universality and stability in the plaque in ultrasound.

Automatic multi-plaque tracking and segmentation in ultrasonic videos

Carotid plaque tracking and segmentation in ultrasound videos is the premise for subsequent plaque property evaluation and treatment plan development. However, the task is quite challenging, as it needs to address the problems of poor image quality, plaque shape variations among frames, the existence of multiple plaques, etc. To overcome these challenges, we propose a new automatic multi-plaque tracking and segmentation (AMPTS) framework. AMPTS consists of three modules. The first module is a multi-object detector, in which a Dual Attention U-Net is proposed to detect multiple plaques and vessels simultaneously. The second module is a set of single-object trackers that can utilize the previous tracking results efficiently and achieve stable tracking of the current target by using channel attention and a ranking strategy. To make the first module and the second module work together, a parallel tracking module based on a simplified 'tracking-by-detection' mechanism is proposed to solve the challenge of tracking object variation. Extensive experiments are conducted to compare the proposed method with several state-of-the-art deep learning based methods. The experimental results demonstrate that the proposed method has high accuracy and generalizability with a Dice similarity coefficient of 0.83 which is 0.16, 0.06 and 0.27 greater than MAST (Lai et al., 2020), Track R-CNN (Voigtlaender et al., 2019) and VSD (Yang et al., 2019) respectively and has made significant improvements on seven other indicators. In the additional Testing set 2, our method achieved a Dice similarity coefficient of 0.80, an accuracy of 0.79, a precision of 0.91, a Recall 0.70, a F1 score of 0.79, an AP@0.5 of 0.92, an AP@0.7 of 0.74, and an expected average overlap of 0.79. Numerous ablation studies suggest the effectiveness of each proposed component and the great potential for multiple carotid plaques tracking and segmentation in clinical practice.

Multimodality carotid plaque tissue characterization and classification in the artificial intelligence paradigm: a narrative review for stroke application

Cardiovascular disease (CVD) is one of the leading causes of morbidity and mortality in the United States of America and globally. Carotid arterial plaque, a cause and also a marker of such CVD, can be detected by various non-invasive imaging modalities such as magnetic resonance imaging (MRI), computer tomography (CT), and ultrasound (US). Characterization and classification of carotid plaque-type in these imaging modalities, especially into symptomatic and asymptomatic plaque, helps in the planning of carotid endarterectomy or stenting. It can be challenging to characterize plaque components due to (I) partial volume effect in magnetic resonance imaging (MRI) or (II) varying Hausdorff values in plaque regions in CT, and (III) attenuation of echoes reflected by the plaque during US causing acoustic shadowing. Artificial intelligence (AI) methods have become an indispensable part of healthcare and their applications to the non-invasive imaging technologies such as MRI, CT, and the US. In this narrative review, three main types of AI models (machine learning, deep learning, and transfer learning) are analyzed when applied to MRI, CT, and the US. A link between carotid plaque characteristics and the risk of coronary artery disease is presented. With regard to characterization, we review tools and techniques that use AI models to distinguish carotid plaque types based on signal processing and feature strengths. We conclude that AI-based solutions offer an accurate and robust path for tissue characterization and classification for carotid artery plaque imaging in all three imaging modalities. Due to cost, user-friendliness, and clinical effectiveness, AI in the US has dominated the most.

Multicenter Computer-Aided Diagnosis for Lymph Nodes Using Unsupervised Domain-Adaptation Networks Based on Cross-Domain Confounding Representations

To achieve the robust high-performance computer-aided diagnosis systems for lymph nodes, CT images may be typically collected from multicenter data, which cause the isolated performance of the model based on different data source centers. The variability adaptation problem of lymph node data which is related to the problem of domain adaptation in deep learning differs from the general domain adaptation problem because of the typically larger CT image size and more complex data distributions. Therefore, domain adaptation for this problem needs to consider the shared feature representation and even the conditioning information of each domain so that the adaptation network can capture significant discriminative representations in a domain-invariant space. This paper extracts domain-invariant features based on a cross-domain confounding representation and proposes a cycle-consistency learning framework to encourage the network to preserve class-conditioning information through cross-domain image translations. Compared with the performance of different domain adaptation methods, the accurate rate of our method achieves at least 4.4% points higher under multicenter lymph node data. The pixel-level cross-domain image mapping and the semantic-level cycle consistency provided a stable confounding representation with class-conditioning information to achieve effective domain adaptation under complex feature distribution.

Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis

Machine learning (ML) algorithms have made a tremendous impact in the field of medical imaging. While medical imaging datasets have been growing in size, a challenge for supervised ML algorithms that is frequently mentioned is the lack of annotated data. As a result, various methods that can learn with less/other types of supervision, have been proposed. We give an overview of semi-supervised, multiple instance, and transfer learning in medical imaging, both in diagnosis or segmentation tasks. We also discuss connections between these learning scenarios, and opportunities for future research. A dataset with the details of the surveyed papers is available via https://figshare.com/articles/Database_of_surveyed_literature_in_Not-so-supervised_a_survey_of_semi-supervised_multi-instance_and_transfer_learning_in_medical_image_analysis_/7479416.

Transfer Learning for Image Segmentation by Combining Image Weighting and Kernel Learning

Many medical image segmentation methods are based on the supervised classification of voxels. Such methods generally perform well when provided with a training set that is representative of the test images to the segment. However, problems may arise when training and test data follow different distributions, for example, due to differences in scanners, scanning protocols, or patient groups. Under such conditions, weighting training images according to distribution similarity have been shown to greatly improve performance. However, this assumes that a part of the training data is representative of the test data; it does not make unrepresentative data more similar. We, therefore, investigate kernel learning as a way to reduce differences between training and test data and explore the added value of kernel learning for image weighting. We also propose a new image weighting method that minimizes maximum mean discrepancy (MMD) between training and test data, which enables the joint optimization of image weights and kernel. Experiments on brain tissue, white matter lesion, and hippocampus segmentation show that both kernel learning and image weighting, when used separately, greatly improve performance on heterogeneous data. Here, MMD weighting obtains similar performance to previously proposed image weighting methods. Combining image weighting and kernel learning, optimized either individually or jointly, can give a small additional improvement in performance.

Automatic Methods for Carotid Contrast-Enhanced Ultrasound Imaging Quantification of Adventitial Vasa Vasorum

Adventitial vasa vasorum are physiologic microvessels that nourish artery walls. In the presence of cardiovascular risk factors, these microvessels proliferate abnormally. Studies have reported that they are the first stage of atheromatous disease. Contrast-enhanced ultrasound (CEUS) of the carotid allows direct, quantitative and non-invasive visualization of the adventitial vasa vasorum. Hence, the development of computer-assisted methods that speed image analysis and eliminate user subjectivity is important. We developed methods for automatic analyses and quantification of vasa vasorum neovascularization in CEUS and tested these methods in a cohort of 186 individuals, 63 of whom were healthy volunteers. We implemented alternative automatic strategies for using the images to stratify patients according to their risk group and compare the strategies with respect to diagnostic performance. An automatic single-parameter strategy performs less effectively than the corresponding Arcidiacono method based on manual interpretation of the images (68 < area under the receiver operating characteristic curve [AUROC] for the manual Arcidiacono method < 82; 60 < AUROC for the automatic single-parameter strategy < 63). However, by use of additional image parameters, an automatic multiparameter strategy has significantly improved performance with respect to the manual Arcidiacono method (78 < AUROC < 90). The automatic multiparameter strategy is a valuable alternative to the manual Arcidiacono method, improving both diagnostic speed and diagnostic accuracy.

Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training

To realize the full potential of deep learning for medical imaging, large annotated datasets are required for training. Such datasets are difficult to acquire due to privacy issues, lack of experts available for annotation, underrepresentation of rare conditions, and poor standardization. The lack of annotated data has been addressed in conventional vision applications using synthetic images refined via unsupervised adversarial training to look like real images. However, this approach is difficult to extend to general medical imaging because of the complex and diverse set of features found in real human tissues. We propose a novel framework that uses a reverse flow, where adversarial training is used to make real medical images more like synthetic images, and clinically-relevant features are preserved via self-regularization. These domain-adapted synthetic-like images can then be accurately interpreted by networks trained on large datasets of synthetic medical images. We implement this approach on the notoriously difficult task of depth-estimation from monocular endoscopy which has a variety of applications in colonoscopy, robotic surgery, and invasive endoscopic procedures. We train a depth estimator on a large data set of synthetic images generated using an accurate forward model of an endoscope and an anatomically-realistic colon. Our analysis demonstrates that the structural similarity of endoscopy depth estimation in a real pig colon predicted from a network trained solely on synthetic data improved by 78.7% by using reverse domain adaptation.

Noninvasive Vascular Modulography Method for Imaging the Local Elasticity of Atherosclerotic Plaques: Simulation and In Vitro Vessel Phantom Study

Mechanical and morphological characterization of atherosclerotic lesions in carotid arteries remains an essential step for the evaluation of rupture prone plaques and the prevention of strokes. In this paper, we propose a noninvasive vascular imaging modulography (NIV-iMod) method, which is capable of reconstructing a heterogeneous Young's modulus distribution of a carotid plaque from the Von Mises strain elastogram. Elastograms were computed with noninvasive ultrasound images using the Lagrangian speckle model estimator and a dynamic segmentation-optimization procedure to highlight mechanical heterogeneities. This methodology, based on continuum mechanics, was validated in silico with finite-element model strain fields and ultrasound simulations, and in vitro with polyvinyl alcohol cryogel phantoms based on magnetic resonance imaging geometries of carotid plaques. In silico, our results show that the NiV-iMod method: 1) successfully detected and quantified necrotic core inclusions with high positive predictive value (PPV) and sensitivity value (SV) of 81±10% and 91±6%; 2) quantified Young's moduli of necrotic cores, fibrous tissues, and calcium inclusions with mean values of 32±23, 515±30, and 3160±218 kPa (ground true values are 10, 600, and 5000 kPa); and 3) overestimated the cap thickness by . In vitro, the PPV and SV for detecting soft inclusions were 60±21% and 88±9%, and Young's modulus mean values of mimicking lipid, fibrosis, and calcium were 34±19, 193±14, and 649±118 kPa (ground true values are 25±3, 182±21, and 757±87 kPa).

Quantitative Analysis of Lipid-Rich Necrotic Core in Carotid Atherosclerotic Plaques by In Vivo Magnetic Resonance Imaging and Clinical Outcomes

Background

The aim of this study was to explore the accuracy of in vivo magnetic resonance imaging (MRI) in the quantitative evaluation of lipid-rich necrotic core (LRNC) in carotid atherosclerotic plaques compared with histopathology, and to assess the association of LRNC size with cerebral ischemia symptoms.

Material/Methods

Thirty patients were enrolled and 19 patients (16 men and 3 women) were analyzed. All the patients were submitted to MRI on a Siemens Avanto (1.5-Tesla) device before carotid endarterectomy (CEA). The scanning protocol included three-dimensional time of flight (3D TOF), T₁-weighted image (T₁WI), T₂-weighted image (T₂WI), turbo spin-echo T₂-weighted (T₂-TSE), and contrast-enhanced T₁-weighted image. MRI images were reviewed for quantitative measurements of LRNC areas. LRNC specimens were collected for histology. Percentages of LRNC area to total vessel area were assessed to determine the association of MRI with histological findings.

Results

There were 151 pairs of matched MRI and pathological sections. LRNC area percentages (LRNC area/vessel area) measured by MRI and histology were 20.6±9.0% and 18.7±9.5%, respectively (r=0.69, p<0.001). Twelve out of 19 patients had symptoms (S-group; 3 had recent stroke, 3 had a recent stroke and a history of transient ischemic attack (TIA), and 6 had TIA); the remaining 7 subjects showed no symptoms (NS-group). LRNC area percentages in the S- and NS-groups were 22.2±5.8% and 12.6±10.7%, respectively (p<0.05).

Conclusions

MRI can quantitatively measure LRNC in carotid atherosclerotic plaques, and may be useful in predicting the rupture risk of plaques. These findings provide a basis for imaging use in individualized treatment plan.

Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection

PURPOSE

We present a method for prostate cancer (PCa) detection using temporal enhanced ultrasound (TeUS) data obtained either from radiofrequency (RF) ultrasound signals or B-mode images.

METHODS

For the first time, we demonstrate that by applying domain adaptation and transfer learning methods, a tissue classification model trained on TeUS RF data (source domain) can be deployed for classification using TeUS B-mode data alone (target domain), where both data are obtained on the same ultrasound scanner. This is a critical step for clinical translation of tissue classification techniques that primarily rely on accessing RF data, since this imaging modality is not readily available on all commercial scanners in clinics. Proof of concept is provided for in vivo characterization of PCa using TeUS B-mode data, where different nonlinear processing filters in the pipeline of the RF to B-mode conversion result in a distribution shift between the two domains.

RESULTS

Our in vivo study includes data obtained in MRI-guided targeted procedure for prostate biopsy. We achieve comparable area under the curve using TeUS RF and B-mode data for medium to large cancer tumor sizes in biopsy cores (>4 mm).

CONCLUSION

Our result suggests that the proposed adaptation technique is successful in reducing the divergence between TeUS RF and B-mode data.

Machine learning approaches in medical image analysis: From detection to diagnosis

Machine learning approaches are increasingly successful in image-based diagnosis, disease prognosis, and risk assessment. This paper highlights new research directions and discusses three main challenges related to machine learning in medical imaging: coping with variation in imaging protocols, learning from weak labels, and interpretation and evaluation of results.