The Radon Cumulative Distribution Transform and Its Application to Image Classification

Recent Advances in Optimal Transport for Machine Learning

Recently, Optimal Transport has been proposed as a probabilistic framework in Machine Learning for comparing and manipulating probability distributions. This is rooted in its rich history and theory, and has offered new solutions to different problems in machine learning, such as generative modeling and transfer learning. In this survey we explore contributions of Optimal Transport for Machine Learning over the period 2012 - 2023, focusing on four sub-fields of Machine Learning: supervised, unsupervised, transfer and reinforcement learning. We further highlight the recent development in computational Optimal Transport and its extensions, such as partial, unbalanced, Gromov and Neural Optimal Transport, and its interplay with Machine Learning practice.

Impact of GAN artifacts for simulating mammograms on identifying mammographically occult cancer

Purpose

Generative adversarial networks (GANs) can synthesize various feasible-looking images. We showed that a GAN, specifically a conditional GAN (CGAN), can simulate breast mammograms with normal, healthy appearances and can help detect mammographically-occult (MO) cancer. However, similar to other GANs, CGANs can suffer from various artifacts, e.g., checkerboard artifacts, that may impact the quality of the final synthesized image, as well as the performance of detecting MO cancer. We explored the types of GAN artifacts that exist in mammogram simulations and their effect on MO cancer detection.

Approach

We first trained a CGAN using digital mammograms (FFDMs) of 1366 women with normal/healthy breasts. Then, we tested the trained CGAN on an independent MO cancer dataset with 333 women with dense breasts (97 MO cancers). We trained a convolutional neural network (CNN) on the MO cancer dataset, in which real and simulated mammograms were fused, to identify women with MO cancer. Then, a radiologist who was independent of the development of the CGAN algorithms evaluated the entire MO cancer dataset to identify and annotate artifacts in the simulated mammograms.

Results

We found four artifact types, including checkerboard, breast boundary, nipple-areola complex, and black spots around calcification artifacts, with an overall incidence rate over 69% (the individual incident rate ranged from 9% to 53%) from both normal and MO cancer samples. We then evaluated their potential impact on MO cancer detection. Even though various artifacts existed in the simulated mammogram, we found that it still provided complementary information for MO cancer detection when it was combined with the real mammograms.

Conclusions

We found that artifacts were pervasive in the CGAN-simulated mammograms. However, they did not negatively affect our MO cancer detection algorithm; the simulated mammograms still provided complementary information for MO cancer detection when combined with real mammograms.

Unsupervised domain adaptation for Covid-19 classification based on balanced slice Wasserstein distance

Covid-19 has swept the world since 2020, taking millions of lives. In order to seek a rapid diagnosis of Covid-19, deep learning-based Covid-19 classification methods have been extensively developed. However, deep learning relies on many samples with high-quality labels, which is expensive. To this end, we propose a novel unsupervised domain adaptation method to process many different but related Covid-19 X-ray images. Unlike existing unsupervised domain adaptation methods that cannot handle conditional class distributions, we adopt a balanced Slice Wasserstein distance as the metric for unsupervised domain adaptation to solve this problem. Multiple standard datasets for domain adaptation and X-ray datasets of different Covid-19 are adopted to verify the effectiveness of our proposed method. Experimented by cross-adopting multiple datasets as source and target domains, respectively, our proposed method can effectively capture discriminative and domain-invariant representations with better data distribution matching.

Invariance encoding in sliced-Wasserstein space for image classification with limited training data

Deep convolutional neural networks (CNNs) are broadly considered to be state-of-the-art generic end-to-end image classification systems. However, they are known to underperform when training data are limited and thus require data augmentation strategies that render the method computationally expensive and not always effective. Rather than using a data augmentation strategy to encode invariances as typically done in machine learning, here we propose to mathematically augment a nearest subspace classification model in sliced-Wasserstein space by exploiting certain mathematical properties of the Radon Cumulative Distribution Transform (R-CDT), a recently introduced image transform. We demonstrate that for a particular type of learning problem, our mathematical solution has advantages over data augmentation with deep CNNs in terms of classification accuracy and computational complexity, and is particularly effective under a limited training data setting. The method is simple, effective, computationally efficient, non-iterative, and requires no parameters to be tuned. Python code implementing our method is available at https://github.com/rohdelab/mathematical augmentation. Our method is integrated as a part of the software package PyTransKit, which is available at https://github.com/rohdelab/PyTransKit.

Predicting Malignancy of Breast Imaging Findings Using Quantitative Analysis of Contrast-Enhanced Mammography (CEM)

We sought to develop new quantitative approaches to characterize the spatial distribution of mammographic density and contrast enhancement of suspicious contrast-enhanced mammography (CEM) findings to improve malignant vs. benign classifications of breast lesions. We retrospectively analyzed all breast lesions that underwent CEM imaging and tissue sampling at our institution from 2014–2020 in this IRB-approved study. A penalized linear discriminant analysis was used to classify lesions based on the averaged histograms of radial distributions of mammographic density and contrast enhancement. T-tests were used to compare the classification accuracies of density, contrast, and concatenated density and contrast histograms. Logistic regression and AUC-ROC analyses were used to assess if adding demographic and clinical data improved the model accuracy. A total of 159 suspicious findings were evaluated. Density histograms were more accurate in classifying lesions as malignant or benign than a random classifier (62.37% vs. 48%; p < 0.001), but the concatenated density and contrast histograms demonstrated a higher accuracy (71.25%; p < 0.001) than the density histograms alone. Including the demographic and clinical data in our models led to a higher AUC-ROC than concatenated density and contrast images (0.81 vs. 0.70; p < 0.001). In the classification of invasive vs. non-invasive malignancy, the concatenated density and contrast histograms demonstrated no significant improvement in accuracy over the density histograms alone (77.63% vs. 78.59%; p = 0.504). Our findings suggest that quantitative differences in the radial distribution of mammographic density could be used to discriminate malignant from benign breast findings; however, classification accuracy was significantly improved with the addition of contrast-enhanced imaging data from CEM. Adding patient demographic and clinical information further improved the classification accuracy.

On a Linear Gromov-Wasserstein Distance

Gromov-Wasserstein distances are generalization of Wasserstein distances, which are invariant under distance preserving transformations. Although a simplified version of optimal transport in Wasserstein spaces, called linear optimal transport (LOT), was successfully used in practice, there does not exist a notion of linear Gromov-Wasserstein distances so far. In this paper, we propose a definition of linear Gromov-Wasserstein distances. We motivate our approach by a generalized LOT model, which is based on barycentric projection maps of transport plans. Numerical examples illustrate that the linear Gromov-Wasserstein distances, similarly as LOT, can replace the expensive computation of pairwise Gromov-Wasserstein distances in applications like shape classification.

Unsupervised domain adaptation of bearing fault diagnosis based on Join Sliced Wasserstein Distance

Deep neural networks have been successfully utilized in the mechanical fault diagnosis, however, a large number of them have been based on the same assumption that training and test datasets followed the same distributions. Unfortunately, the mechanical systems are easily affected by environment noise interference, speed or load change. Consequently, the trained networks have poor generalization under various working conditions. Recently, unsupervised domain adaptation has been concentrated on more and more attention since it can handle different but related data. Sliced Wasserstein Distance has been successfully utilized in unsupervised domain adaptation and obtained excellent performances. However, most of the approaches have ignored the class conditional distribution. In this paper, a novel approach named Join Sliced Wasserstein Distance (JSWD) has been proposed to address the above issue. Four bearing datasets have been selected to validate the practicability and effectiveness of the JSWD framework. The experimental results have demonstrated that about 5% accuracy is improved by JSWD with consideration of the conditional probability than no the conditional probability, in addition, the other experimental results have indicated that JSWD could effectively capture the distinguishable and domain-invariant representations and have a has superior data distribution matching than the previous methods under various application scenarios.

An Optimal Transport Based Global Similarity Index for Remote Sensing Products Comparison

Remote sensing products, such as land cover data products, are essential for a wide range of scientific studies and applications, and their quality evaluation and relative comparison have become a major issue that needs to be studied. Traditional methods, such as error matrices, are not effective in describing spatial distribution because they are based on a pixel-by-pixel comparison. In this paper, the relative quality comparison of two remote sensing products is turned into the difference measurement between the spatial distribution of pixels by proposing a max-sliced Wasserstein distance-based similarity index. According to optimal transport theory, the mathematical expression of the proposed similarity index is firstly clarified, and then its rationality is illustrated, and finally, experiments on three open land cover products (GLCFCS30, FROMGLC, CNLUCC) are conducted. Results show that based on this proposed similarity index-based relative quality comparison method, the spatial difference, including geometric shapes and spatial locations between two different remote sensing products in raster form, can be quantified. The method is particularly useful in cases where there exists misregistration between datasets, while pixel-based methods will lose their robustness.

Identifying Women With Mammographically- Occult Breast Cancer Leveraging GAN-Simulated Mammograms

Our objective is to show the feasibility of using simulated mammograms to detect mammographically-occult (MO) cancer in women with dense breasts and a normal screening mammogram who could be triaged for additional screening with magnetic resonance imaging (MRI) or ultrasound. We developed a Conditional Generative Adversarial Network (CGAN) to simulate a mammogram with normal appearance using the opposite mammogram as the condition. We used a Convolutional Neural Network (CNN) trained on Radon Cumulative Distribution Transform (RCDT) processed mammograms to detect MO cancer. For training CGAN, we used screening mammograms of 1366 women. For MO cancer detection, we used screening mammograms of 333 women (97 MO cancer) with dense breasts. We simulated the right mammogram for normal controls and the cancer side for MO cancer cases. We created two RCDT images, one from a real mammogram pair and another from a real-simulated mammogram pair. We finetuned a VGG16 on resulting RCDT images to classify the women with MO cancer. We compared the classification performance of the CNN trained on fused RCDT images, CNN_Fused to that of trained only on real RCDT images, CNN_Real, and to that of trained only on simulated RCDT images, CNN_Simulated. The test AUC for CNN_Fused was 0.77 with a 95% confidence interval (95CI) of [0.71, 0.83], which was statistically better (p-value < 0.02) than the CNN_Real AUC of 0.70 with a 95CI of [0.64, 0.77] and CNN_Simulated AUC of 0.68 with a 95CI of [0.62, 0.75]. It showed that CGAN simulated mammograms can help MO cancer detection.

Radon Cumulative Distribution Transform Subspace Modeling for Image Classification

We present a new supervised image classification method applicable to a broad class of image deformation models. The method makes use of the previously described Radon Cumulative Distribution Transform (R-CDT) for image data, whose mathematical properties are exploited to express the image data in a form that is more suitable for machine learning. While certain operations such as translation, scaling, and higher-order transformations are challenging to model in native image space, we show the R-CDT can capture some of these variations and thus render the associated image classification problems easier to solve. The method - utilizing a nearest-subspace algorithm in the R-CDT space - is simple to implement, non-iterative, has no hyper-parameters to tune, is computationally efficient, label efficient, and provides competitive accuracies to state-of-the-art neural networks for many types of classification problems. In addition to the test accuracy performances, we show improvements (with respect to neural network-based methods) in terms of computational efficiency (it can be implemented without the use of GPUs), number of training samples needed for training, as well as out-of-distribution generalization. The Python code for reproducing our results is available at [1].

Transport-based pattern recognition versus deep neural networks in underwater OAM communications

Comparisons between machine learning and optimal transport-based approaches in classifying images are made in underwater orbital angular momentum (OAM) communications. A model is derived that justifies optimal transport for use in attenuated water environments. OAM pattern demultiplexing is performed using optimal transport and deep neural networks and compared to each other. Additionally, some of the complications introduced by signal attenuation are highlighted. The Radon cumulative distribution transform (R-CDT) is applied to OAM patterns to transform them to a linear subspace. The original OAM images and the R-CDT transformed patterns are used in several classification algorithms, and results are compared. The selected classification algorithms are the nearest subspace algorithm, a shallow convolutional neural network (CNN), and a deep neural network. It is shown that the R-CDT transformed images are more accurate than the original OAM images in pattern classification. Also, the nearest subspace algorithm performs better than the selected CNNs in OAM pattern classification in underwater environments.

PARTITIONING SIGNAL CLASSES USING TRANSPORT TRANSFORMS FOR DATA ANALYSIS AND MACHINE LEARNING

A relatively new set of transport-based transforms (CDT, R-CDT, LOT) have shown their strength and great potential in various image and data processing tasks such as parametric signal estimation, classification, cancer detection among many others. It is hence worthwhile to elucidate some of the mathematical properties that explain the successes of these transforms when they are used as tools in data analysis, signal processing or data classification. In particular, we give conditions under which classes of signals that are created by algebraic generative models are transformed into convex sets by the transport transforms. Such convexification of the classes simplify the classification and other data analysis and processing problems when viewed in the transform domain. More specifically, we study the extent and limitation of the convexification ability of these transforms under an algebraic generative modeling framework. We hope that this paper will serve as an introduction to these transforms and will encourage mathematicians and other researchers to further explore the theoretical underpinnings and algorithmic tools that will help understand the successes of these transforms and lay the groundwork for further successful applications.

Modulation format identification of optical signals: an approach based on singular value decomposition of Stokes space projections

In this paper, two Stokes space (SS) analysis schemes for modulation format identification (MFI) are proposed. These schemes are based on singular value decomposition (SVD) and Radon transform (RT) for feature extraction. The singular values (SVs) are extracted from the SS projections for different modulation formats to discriminate between them. The SS projections are obtained at different optical signal-to-noise ratios (OSNRs) ranging from 11 to 30 dB for seven dual-polarized modulation formats. The first scheme depends on the SVDs of the SS projections on three planes, while the second scheme depends on the SVDs of the RTs of the SS projections. Different classifiers including support vector machine (SVM), decision tree (DT), and K-nearest neighbor (KNN) for MFI based on the obtained features are used. Both simulation and experimental setups are arranged and tested for proof of concept of the proposed schemes for the MFI task. Complexity reduction is studied for the SVD scheme by applying the decimation of the projections by two and four to achieve an acceptable classification rate, while reducing the computation time. Also, the effect of the variation of phase noise (PN) and state of polarization (SoP) on the accuracy of the MFI is considered at all OSNRs. The two proposed schemes are capable of identifying the polarization multiplexed modulation formats blindly with high accuracy levels up to 98%, even at low OSNR values of 12 dB, high PN levels up to 10 MHz, and SoP up to 45°.

Cell Image Classification: A Comparative Overview

Cell image classification methods are currently being used in numerous applications in cell biology and medicine. Applications include understanding the effects of genes and drugs in screening experiments, understanding the role and subcellular localization of different proteins, as well as diagnosis and prognosis of cancer from images acquired using cytological and histological techniques. The article also reviews three main approaches for cell image classification most often used: numerical feature extraction, end-to-end classification with neural networks (NNs), and transport-based morphometry (TBM). In addition, we provide comparisons on four different cell imaging datasets to highlight the relative strength of each method. The results computed using four publicly available datasets show that numerical features tend to carry the best discriminative information for most of the classification tasks. Results also show that NN-based methods produce state-of-the-art results in the dataset that contains a relatively large number of training samples. Data augmentation or the choice of a more recently reported architecture does not necessarily improve the classification performance of NNs in the datasets with limited number of training samples. If understanding and visualization are desired aspects, TBM methods can offer the ability to invert classification functions, and thus can aid in the interpretation of results. These and other comparison outcomes are discussed with the aim of clarifying the advantages and disadvantages of each method. © 2020 International Society for Advancement of Cytometry.

Detecting mammographically occult cancer in women with dense breasts using deep convolutional neural network and Radon Cumulative Distribution Transform

We have applied the Radon Cumulative Distribution Transform (RCDT) as an image transformation to highlight the subtle difference between left and right mammograms to detect mammographically occult (MO) cancer in women with dense breasts and negative screening mammograms. We developed deep convolutional neural networks (CNNs) as classifiers for estimating the probability of having MO cancer. We acquired screening mammograms of 333 women (97 unilateral MO cancer) with dense breasts and at least two consecutive mammograms and used the immediate prior mammograms, which radiologists interpreted as negative. We used fivefold cross validation to divide our dataset into a training and independent test sets with ratios of 0.8:0.2. We set aside 10% of the training set as a validation set. We applied RCDT on the left and right mammograms of each view. We applied inverse Radon transform to represent the resulting RCDT images in the image domain. We then fine-tuned a VGG16 network pretrained on ImageNet using the resulting images per each view. The CNNs achieved mean areas under the receiver operating characteristic (AUC) curve of 0.73 (standard error, SE = 0.024) and 0.73 (SE = 0.04) for the craniocaudal and mediolateral oblique views, respectively. We combined the scores from two CNNs by training a logistic regression classifier and it achieved a mean AUC of 0.81 (SE = 0.032). In conclusion, we showed that inverse Radon-transformed RCDT images contain information useful for detecting MO cancers and that deep CNNs could learn such information.

Path-Based Dictionary Augmentation: A Framework for Improving k-Sparse Image Processing

We have previously shown that augmenting orthogonal matching pursuit (OMP) with an additional step in the identification stage of each pursuit iteration yields improved k-sparse reconstruction and denoising performance relative to baseline OMP. At each iteration a "path," or geodesic, is generated between the two dictionary atoms that are most correlated with the residual and from this path a new atom that has a greater correlation to the residual than either of the two bracketing atoms is selected. Here, we provide new computational results illustrating improvements in sparse coding and denoising on canonical datasets using both learned and structured dictionaries. Two methods of constructing a path are investigated for each dictionary type: the Euclidean geodesic formed by a linear combination of the two atoms and the 2-Wasserstein geodesic corresponding to the optimal transport map between the atoms. We prove here the existence of a higher-correlation atom in the Euclidean case under assumptions on the two bracketing atoms and introduce algorithmic modifications to improve the likelihood that the bracketing atoms meet those conditions. Although we demonstrate our augmentation on OMP alone, in general it may be applied to any reconstruction algorithm that relies on the selection and sorting of high-similarity atoms during an analysis or identification phase.

Learning to Predict Ischemic Stroke Growth on Acute CT Perfusion Data by Interpolating Low-Dimensional Shape Representations

Cerebrovascular diseases, in particular ischemic stroke, are one of the leading global causes of death in developed countries. Perfusion CT and/or MRI are ideal imaging modalities for characterizing affected ischemic tissue in the hyper-acute phase. If infarct growth over time could be predicted accurately from functional acute imaging protocols together with advanced machine-learning based image analysis, the expected benefits of treatment options could be better weighted against potential risks. The quality of the outcome prediction by convolutional neural networks (CNNs) is so far limited, which indicates that even highly complex deep learning algorithms are not fully capable of directly learning physiological principles of tissue salvation through weak supervision due to a lack of data (e.g., follow-up segmentation). In this work, we address these current shortcomings by explicitly taking into account clinical expert knowledge in the form of segmentations of the core and its surrounding penumbra in acute CT perfusion images (CTP), that are trained to be represented in a low-dimensional non-linear shape space. Employing a multi-scale CNN (U-Net) together with a convolutional auto-encoder, we predict lesion tissue probabilities for new patients. The predictions are physiologically constrained to a shape embedding that encodes a continuous progression between the core and penumbra extents. The comparison to a simple interpolation in the original voxel space and an unconstrained CNN shows that the use of such a shape space can be advantageous to predict time-dependent growth of stroke lesions on acute perfusion data, yielding a Dice score overlap of 0.46 for predictions from expert segmentations of core and penumbra. Our interpolation method models monotone infarct growth robustly on a linear time scale to automatically predict clinically plausible tissue outcomes that may serve as a basis for more clinical measures such as the expected lesion volume increase and can support the decision making on treatment options and triage.

De-multiplexing vortex modes in optical communications using transport-based pattern recognition

Free space optical communications utilizing orbital angular momentum beams have recently emerged as a new technique for communications with potential for increased channel capacity. Turbulence due to changes in the index of refraction emanating from temperature, humidity, and air flow patterns, however, add nonlinear effects to the received patterns, thus making the demultiplexing task more difficult. Deep learning techniques have been previously been applied to solve the demultiplexing problem as an image classification task. Here we make use of a newly developed theory suggesting a link between image turbulence and photon transport through the continuity equation to describe a method that utilizes a "shallow" learning method instead. The decoding technique is tested and compared against previous approaches using deep convolutional neural networks. Results show that the new method can obtain similar classification accuracies (bit error ratio) at a small fraction (1/90) of the computational cost, thus enabling higher bit rates.

Discovery and visualization of structural biomarkers from MRI using transport-based morphometry

Disease in the brain is often associated with subtle, spatially diffuse, or complex tissue changes that may lie beneath the level of gross visual inspection, even on magnetic resonance imaging (MRI). Unfortunately, current computer-assisted approaches that examine pre-specified features, whether anatomically-defined (i.e. thalamic volume, cortical thickness) or based on pixelwise comparison (i.e. deformation-based methods), are prone to missing a vast array of physical changes that are not well-encapsulated by these metrics. In this paper, we have developed a technique for automated pattern analysis that can fully determine the relationship between brain structure and observable phenotype without requiring any a priori features. Our technique, called transport-based morphometry (TBM), is an image transformation that maps brain images losslessly to a domain where they become much more separable. The new approach is validated on structural brain images of healthy older adult subjects where even linear models for discrimination, regression, and blind source separation enable TBM to independently discover the characteristic changes of aging and highlight potential mechanisms by which aerobic fitness may mediate brain health later in life. TBM is a generative approach that can provide visualization of physically meaningful shifts in tissue distribution through inverse transformation. The proposed framework is a powerful technique that can potentially elucidate genotype-structural-behavioral associations in myriad diseases.

A Transportation Lp Distance for Signal Analysis

Transport based distances, such as the Wasserstein distance and earth mover'sdistance, have been shown to be an effective tool in signal and image analysis. The success of transport based distances is in part due to their Lagrangian nature which allows it to capture the important variations in many signal classes. However these distances require the signal to be nonnegative and normalized. Furthermore, the signals are considered as measures and compared by redistributing (transporting) them, which does not directly take into account the signal intensity. Here we study a transport-based distance, called the TLp distance, that combines Lagrangian and intensity modelling and is directly applicable to general, non-positive and multi-channelled signals. The distance can be computed by existing numerical methods. We give an overview of the basic properties of this distance and applications to classification, with multi-channelled non-positive one-dimensional signals and two-dimensional images, and color transfer.

Optimal Mass Transport: Signal processing and machine-learning applications

Transport-based techniques for signal and data analysis have received increased attention recently. Given their ability to provide accurate generative models for signal intensities and other data distributions, they have been used in a variety of applications including content-based retrieval, cancer detection, image super-resolution, and statistical machine learning, to name a few, and shown to produce state of the art results in several applications. Moreover, the geometric characteristics of transport-related metrics have inspired new kinds of algorithms for interpreting the meaning of data distributions. Here we provide a practical overview of the mathematical underpinnings of mass transport-related methods, including numerical implementation, as well as a review, with demonstrations, of several applications. Software accompanying this tutorial is available at [43].