A continuous linear optimal transport approach for pattern analysis in image datasets

Wasserstein task embedding for measuring task similarities

Measuring similarities between different tasks is critical in a broad spectrum of machine learning problems, including transfer, multi-task, continual, and meta-learning. Most current approaches to measuring task similarities are architecture-dependent: (1) relying on pre-trained models, or (2) training networks on tasks and using forward transfer as a proxy for task similarity. In this paper, we leverage the optimal transport theory and define a novel task embedding for supervised classification that is model-agnostic, training-free, and capable of handling (partially) disjoint label sets. In short, given a dataset with ground-truth labels, we perform a label embedding through multi-dimensional scaling and concatenate dataset samples with their corresponding label embeddings. Then, we define the distance between two datasets as the 2-Wasserstein distance between their updated samples. Lastly, we leverage the 2-Wasserstein embedding framework to embed tasks into a vector space in which the Euclidean distance between the embedded points approximates the proposed 2-Wasserstein distance between tasks. We show that the proposed embedding leads to a significantly faster comparison of tasks compared to related approaches like the Optimal Transport Dataset Distance (OTDD). Furthermore, we demonstrate the effectiveness of our embedding through various numerical experiments and show statistically significant correlations between our proposed distance and the forward and backward transfer among tasks on a wide variety of image recognition datasets.

Discovering the gene-brain-behavior link in autism via generative machine learning

Autism is traditionally diagnosed behaviorally but has a strong genetic basis. A genetics-first approach could transform understanding and treatment of autism. However, isolating the gene-brain-behavior relationship from confounding sources of variability is a challenge. We demonstrate a novel technique, 3D transport-based morphometry (TBM), to extract the structural brain changes linked to genetic copy number variation (CNV) at the 16p11.2 region. We identified two distinct endophenotypes. In data from the Simons Variation in Individuals Project, detection of these endophenotypes enabled 89 to 95% test accuracy in predicting 16p11.2 CNV from brain images alone. Then, TBM enabled direct visualization of the endophenotypes driving accurate prediction, revealing dose-dependent brain changes among deletion and duplication carriers. These endophenotypes are sensitive to articulation disorders and explain a portion of the intelligence quotient variability. Genetic stratification combined with TBM could reveal new brain endophenotypes in many neurodevelopmental disorders, accelerating precision medicine, and understanding of human neurodiversity.

On a Linear Gromov-Wasserstein Distance

Gromov-Wasserstein distances are generalization of Wasserstein distances, which are invariant under distance preserving transformations. Although a simplified version of optimal transport in Wasserstein spaces, called linear optimal transport (LOT), was successfully used in practice, there does not exist a notion of linear Gromov-Wasserstein distances so far. In this paper, we propose a definition of linear Gromov-Wasserstein distances. We motivate our approach by a generalized LOT model, which is based on barycentric projection maps of transport plans. Numerical examples illustrate that the linear Gromov-Wasserstein distances, similarly as LOT, can replace the expensive computation of pairwise Gromov-Wasserstein distances in applications like shape classification.

Investigating impact of cardiorespiratory fitness in reducing brain tissue loss caused by ageing

Mitigating the loss of brain tissue due to age is a major problem for an ageing population. Improving cardiorespiratory fitness has been suggested as a possible strategy, but the influenceon brain morphology has not been fully characterized. To investigate the dependent shifts in brain tissue distribution as a function of cardiorespiratory fitness, we used a 3D transport-based morphometry approach. In this study of 172 inactive older adults aged 58-81 (66.5 ± 5.7) years, cardiorespiratory fitness was determined by VO 2 peak (ml/kg/min) during graded exercise and brain morphology was assessed through structural magnetic resonance imaging. After correcting for covariates including age (in the fitness model), gender and level of education, we compared dependent tissue shifts with age to those due to V O 2   peak . We found a significant association between cardiorespiratory fitness and brain tissue distribution (white matter, r = 0.30, P = 0.003; grey matter, r = 0.40, P < 0.001) facilitated by direct visualization of the brain tissue shifts due to cardiorespiratory fitness through inverse transformation-a key capability of 3D transport-based morphometry. A strong statistical correlation was found between brain tissue changes related to ageing and those associated with lower cardiorespiratory fitness (white matter, r = 0.62, P < 0.001; grey matter, r = 0.74, P < 0.001). In both cases, frontotemporal regions shifted the most while basal ganglia shifted the least. Our results highlight the importance of cardiorespiratory fitness in maintaining brain health later in life. Furthermore, this work demonstrates 3D transport-based morphometry as a novel neuroinformatic technology that may aid assessment of therapeutic approaches for brain ageing and neurodegenerative diseases.

PARTITIONING SIGNAL CLASSES USING TRANSPORT TRANSFORMS FOR DATA ANALYSIS AND MACHINE LEARNING

A relatively new set of transport-based transforms (CDT, R-CDT, LOT) have shown their strength and great potential in various image and data processing tasks such as parametric signal estimation, classification, cancer detection among many others. It is hence worthwhile to elucidate some of the mathematical properties that explain the successes of these transforms when they are used as tools in data analysis, signal processing or data classification. In particular, we give conditions under which classes of signals that are created by algebraic generative models are transformed into convex sets by the transport transforms. Such convexification of the classes simplify the classification and other data analysis and processing problems when viewed in the transform domain. More specifically, we study the extent and limitation of the convexification ability of these transforms under an algebraic generative modeling framework. We hope that this paper will serve as an introduction to these transforms and will encourage mathematicians and other researchers to further explore the theoretical underpinnings and algorithmic tools that will help understand the successes of these transforms and lay the groundwork for further successful applications.

Irregular Respiratory Motion Compensation for Liver Contrast-Enhanced Ultrasound via Transport-Based Motion Estimation

Contrast-enhanced ultrasound (CEUS) imaging has been widely applied for the detection and characterization of focal liver lesions (FLLs), for its ability to visualize the blood flow in real time. However, cyclic liver motion poses a great challenge to the recovery of perfusion curves as well as quantitative kinetic parameters estimation. Recently, a few gating methods have been proposed to eliminate unexpected intensity fluctuations by the breathing phase estimation, with the assumption that each breathing phase corresponds to a specific lesion position strictly. While practical liver motion tends to be irregular due to changes in the patient's underlying physiologic status, that is, the same phase might not correspond to the same position. To tackle the challenge of motion irregularity, we present a novel motion estimation-based respiratory compensation method, named RCME, which first estimates salient motion information through the framework of optimal transport (OT) by jointly modeling pixel intensity as well as their locations and then employs sparse subspace clustering (SSC) to identify the subset of frames acquired at the same position. Our proposed method is evaluated on 15 clinical CEUS sequences in both qualitative and quantitative manners. Experimental results demonstrate good performance on irregular liver motion compensation.

Physics approach to the variable-mass optimal-transport problem

Optimal transport (OT) has become a discipline by itself that offers solutions to a wide range of theoretical problems in probability and mathematics with applications in several applied fields such as imaging sciences, machine learning, and in data sciences in general. The traditional OT problem suffers from a severe limitation: its balance condition imposes that the two distributions to be compared be normalized and have the same total mass. However, it is important for many applications to be able to relax this constraint and allow for mass creation and/or destruction. This is true, for example, in all problems requiring partial matching. In this paper, we propose an approach to solving a generalized version of the OT problem, which we refer to as the discrete variable-mass optimal-transport (VMOT) problem, using techniques adapted from statistical physics. Our first contribution is to fully describe this formalism, including all the proofs of its main claims. In particular, we derive a strongly concave effective free-energy function that captures the constraints of the VMOT problem at a finite temperature. From its maximum we derive a weak distance (i.e., a divergence) between possibly unbalanced distribution functions. The temperature-dependent OT distance decreases monotonically to the standard variable-mass OT distance, providing a robust framework for temperature annealing. Our second contribution is to show that the implementation of this formalism has the same properties as the regularized OT algorithms in time complexity, making it a competitive approach to solving the VMOT problem. We illustrate applications of the framework to the problem of partial two- and three-dimensional shape-matching problems.

Enabling early detection of osteoarthritis from presymptomatic cartilage texture maps via transport-based learning

Many diseases have no visual cues in the early stages, eluding image-based detection. Today, osteoarthritis (OA) is detected after bone damage has occurred, at an irreversible stage of the disease. Currently no reliable method exists for OA detection at a reversible stage. We present an approach that enables sensitive OA detection in presymptomatic individuals. Our approach combines optimal mass transport theory with statistical pattern recognition. Eighty-six healthy individuals were selected from the Osteoarthritis Initiative, with no symptoms or visual signs of disease on imaging. On 3-y follow-up, a subset of these individuals had progressed to symptomatic OA. We trained a classifier to differentiate progressors and nonprogressors on baseline cartilage texture maps, which achieved a robust test accuracy of 78% in detecting future symptomatic OA progression 3 y prior to symptoms. This work demonstrates that OA detection may be possible at a potentially reversible stage. A key contribution of our work is direct visualization of the cartilage phenotype defining predictive ability as our technique is generative. We observe early biochemical patterns of fissuring in cartilage that define future onset of OA. In the future, coupling presymptomatic OA detection with emergent clinical therapies could modify the outcome of a disease that costs the United States healthcare system $16.5 billion annually. Furthermore, our technique is broadly applicable to earlier image-based detection of many diseases currently diagnosed at advanced stages today.

Cell Image Classification: A Comparative Overview

Cell image classification methods are currently being used in numerous applications in cell biology and medicine. Applications include understanding the effects of genes and drugs in screening experiments, understanding the role and subcellular localization of different proteins, as well as diagnosis and prognosis of cancer from images acquired using cytological and histological techniques. The article also reviews three main approaches for cell image classification most often used: numerical feature extraction, end-to-end classification with neural networks (NNs), and transport-based morphometry (TBM). In addition, we provide comparisons on four different cell imaging datasets to highlight the relative strength of each method. The results computed using four publicly available datasets show that numerical features tend to carry the best discriminative information for most of the classification tasks. Results also show that NN-based methods produce state-of-the-art results in the dataset that contains a relatively large number of training samples. Data augmentation or the choice of a more recently reported architecture does not necessarily improve the classification performance of NNs in the datasets with limited number of training samples. If understanding and visualization are desired aspects, TBM methods can offer the ability to invert classification functions, and thus can aid in the interpretation of results. These and other comparison outcomes are discussed with the aim of clarifying the advantages and disadvantages of each method. © 2020 International Society for Advancement of Cytometry.

Image-derived models of cell organization changes during differentiation and drug treatments

PC12 cells are a popular model system to study changes driving and accompanying neuronal differentiation. While attention has been paid to changes in transcriptional regulation and protein signaling, much less is known about the changes in organization that accompany PC12 differentiation. Fluorescence microscopy can provide extensive information about these changes, although it is difficult to continuously observe changes over many days of differentiation. We describe a generative model of differentiation-associated changes in cell and nuclear shape and their relationship to mitochondrial distribution constructed from images of different cells at discrete time points. We show that the model accurately represents complex cell and nuclear shapes and learn a regression model that relates cell and nuclear shape to mitochondrial distribution; the predictive accuracy of the model increases during differentiation. Most importantly, we propose a method, based on cell matching and interpolation, to produce realistic simulations of the dynamics of cell differentiation from only static images. We also found that the distribution of cell shapes is hollow: most shapes are very different from the average shape. Finally, we show how the method can be used to model nuclear shape changes of human-induced pluripotent stem cells resulting from drug treatments.

Optimal transport at finite temperature

Optimal transport (OT) has become a discipline by itself that offers solutions to a wide range of theoretical problems in probability and mathematics. Despite its appealing theoretical properties, solving the OT problem involves the resolution of a linear program whose computational cost can quickly become prohibitive whenever the size of the problem exceeds a few hundred points. The recent introduction of entropy regularization, however, has led to the development of fast algorithms for solving an approximate OT problem. The successes of those algorithms have resulted in a popularization of the applications of OT in several applied fields such as imaging sciences and machine learning, and in data sciences in general. Problems remain, however, as to the numerical convergence of those regularized approximations towards the actual OT solution. In addition, the physical meaning of this regularization is unclear. In this paper, we propose an approach to solving the discrete OT problem using techniques adapted from statistical physics. Our first contribution is to fully describe this formalism, including all the proofs of its main claims. In particular we derive a strongly concave effective free energy function that captures the constraints of the optimal transport problem at a finite temperature. Its maximum defines a pseudo distance between the two set of weighted points that are compared, which satisfies the triangular inequalities. The temperature dependent OT pseudo distance decreases monotonically to the standard OT distance, providing a robust framework for temperature annealing. Our second contribution is to show that the implementation of this formalism has the same properties as the regularized OT algorithms in time complexity, making it a competitive approach to solving the OT problem. We illustrate applications of the framework to the problem of protein fold recognition based on sequence information only.

Assessing Postconcussive Reaction Time Using Transport-Based Morphometry of Diffusion Tensor Images

BACKGROUND AND PURPOSE

Cognitive deficits are among the most commonly reported post-concussive symptoms, yet the underlying microstructural injury is poorly understood. Our aim was to discover white matter injury underlying reaction time in mild traumatic brain injury DTI by applying transport-based morphometry.

MATERIALS AND METHODS

In this retrospective study, we performed DTI on 64 postconcussive patients (10-28 years of age; 69% male, 31% female) between January 2006 and March 2013. We measured the reaction time percentile by using Immediate Post-Concussion Assessment and Cognitive Testing. Using the 3D transport-based morphometry technique we developed, we mined fractional anisotropy maps to extract the common microstructural injury associated with reaction time percentile in an automated manner. Permutation testing established statistical significance of the extracted injuries. We visualized the physical substrate responsible for reaction time through inverse transport-based morphometry transformation.

RESULTS

The direction in the transport space most correlated with reaction time was significant after correcting for covariates of age, sex, and time from injury (Pearson r = 0.44, P < .01). Inverting the computed direction using transport-based morphometry illustrates physical shifts in fractional anisotropy in the corpus callosum (increase) and within the optic radiations, corticospinal tracts, and anterior thalamic radiations (decrease) with declining reaction time. The observed shifts are consistent with biologic pathways underlying the visual-spatial interpretation and response-selection aspects of reaction time.

CONCLUSIONS

Transport-based morphometry discovers complex white matter injury underlying postconcussive reaction time in an automated manner. The potential influences of edema and axonal loss are visualized in the visual-spatial interpretation and response-selection pathways. Transport-based morphometry can bridge the gap between brain microstructure and function in diseases in which the structural basis is unknown.

Transport-based model for turbulence-corrupted imagery

A new model for turbulence-corrupted imagery is proposed based on the theory of optimal mass transport. By describing the relationship between photon density and the phase of the traveling wave, and combining it with a least action principle, the model suggests a new class of methods for approximately recovering the solution of the photon density flow created by a turbulent atmosphere. Both coherent and incoherent imagery are used to validate and compare the model to other methods typically used to describe this type of data. Given its superior performance in describing experimental data, the new model suggests new algorithms for a variety of atmospheric imaging and wave propagation applications.

Discovery and visualization of structural biomarkers from MRI using transport-based morphometry

Disease in the brain is often associated with subtle, spatially diffuse, or complex tissue changes that may lie beneath the level of gross visual inspection, even on magnetic resonance imaging (MRI). Unfortunately, current computer-assisted approaches that examine pre-specified features, whether anatomically-defined (i.e. thalamic volume, cortical thickness) or based on pixelwise comparison (i.e. deformation-based methods), are prone to missing a vast array of physical changes that are not well-encapsulated by these metrics. In this paper, we have developed a technique for automated pattern analysis that can fully determine the relationship between brain structure and observable phenotype without requiring any a priori features. Our technique, called transport-based morphometry (TBM), is an image transformation that maps brain images losslessly to a domain where they become much more separable. The new approach is validated on structural brain images of healthy older adult subjects where even linear models for discrimination, regression, and blind source separation enable TBM to independently discover the characteristic changes of aging and highlight potential mechanisms by which aerobic fitness may mediate brain health later in life. TBM is a generative approach that can provide visualization of physically meaningful shifts in tissue distribution through inverse transformation. The proposed framework is a powerful technique that can potentially elucidate genotype-structural-behavioral associations in myriad diseases.

A Transportation Lp Distance for Signal Analysis

Transport based distances, such as the Wasserstein distance and earth mover'sdistance, have been shown to be an effective tool in signal and image analysis. The success of transport based distances is in part due to their Lagrangian nature which allows it to capture the important variations in many signal classes. However these distances require the signal to be nonnegative and normalized. Furthermore, the signals are considered as measures and compared by redistributing (transporting) them, which does not directly take into account the signal intensity. Here we study a transport-based distance, called the TLp distance, that combines Lagrangian and intensity modelling and is directly applicable to general, non-positive and multi-channelled signals. The distance can be computed by existing numerical methods. We give an overview of the basic properties of this distance and applications to classification, with multi-channelled non-positive one-dimensional signals and two-dimensional images, and color transfer.

Optimal Mass Transport: Signal processing and machine-learning applications

Transport-based techniques for signal and data analysis have received increased attention recently. Given their ability to provide accurate generative models for signal intensities and other data distributions, they have been used in a variety of applications including content-based retrieval, cancer detection, image super-resolution, and statistical machine learning, to name a few, and shown to produce state of the art results in several applications. Moreover, the geometric characteristics of transport-related metrics have inspired new kinds of algorithms for interpreting the meaning of data distributions. Here we provide a practical overview of the mathematical underpinnings of mass transport-related methods, including numerical implementation, as well as a review, with demonstrations, of several applications. Software accompanying this tutorial is available at [43].

Predictive Nuclear Chromatin Characteristics of Melanoma and Dysplastic Nevi

Background:

The diagnosis of malignant melanoma (MM) is among the diagnostic challenges pathologists encounter on a routine basis. Melanoma may arise in patients with preexisting dysplastic nevi (DN) and it is still the cause of 1.7% of all cancer-related deaths. Melanomas often have overlapping histological features with DN, especially those with severe dysplasia. Nucleotyping for identifying nuclear textural features can analyze nuclear DNA structure and organization. The aim of this study is to differentiate MM and DN using these methodologies.

Methods:

Dermatopathology slides diagnosed as MM and DN were retrieved. The glass slides were scanned using an Aperio ScanScopeXT at ×40 (0.25 μ/pixel). Whole slide images (WSI) were annotated for nuclei selection. Nuclear features to distinguish between MM and DN based on chromatin distributions were extracted from the WSI. The morphological characteristics for each nucleus were quantified with the optimal transport-based linear embedding in the continuous domain. Label predictions for individual cell nucleus are achieved through a modified version of linear discriminant analysis, coupled with the k-nearest neighbor classifier. Label for an unknown patient was set by the voting strategy with its pertaining cell nuclei.

Results:

Nucleotyping of 139 patient cases of melanoma (n = 67) and DN (n = 72) showed that our method had superior classification accuracy of 81.29%. This is a 6.4% gain in differentiating MM and DN, compared with numerical feature-based method. The distribution differences in nuclei morphology between MM and DN can be visualized with biological interpretation.

Conclusions:

Nucleotyping using quantitative and qualitative analyses may provide enough information for differentiating MM from DN using pixel image data. Our method to segment cell nuclei may offer a practical and inexpensive solution in aiding in the accurate diagnosis of melanoma.

The Radon Cumulative Distribution Transform and Its Application to Image Classification

Invertible image representation methods (transforms) are routinely employed as low-level image processing operations based on which feature extraction and recognition algorithms are developed. Most transforms in current use (e.g., Fourier, wavelet, and so on) are linear transforms and, by themselves, are unable to substantially simplify the representation of image classes for classification. Here, we describe a nonlinear, invertible, low-level image processing transform based on combining the well-known Radon transform for image data, and the 1D cumulative distribution transform proposed earlier. We describe a few of the properties of this new transform, and with both theoretical and experimental results show that it can often render certain problems linearly separable in a transform space.