Fast Multiscale Neighbor Embedding

Coupled retrieval of heavy metal nickel concentration in agricultural soil from spaceborne hyperspectral imagery

Widespread soil contamination endangers public health and undermines global attempts to achieve the United Nations Sustainable Development Goals. Due to the lack of relevant studies and low precision of spaceborne spectroscopy, estimating soil heavy metal concentrations is challenging. In this study, we developed a coupled retrieval to qualify the heavy metal nickel (Ni) concentration in agricultural soil from spaceborne hyperspectral imagery. The retrieval couples spectral feature extraction from multi-scale discrete wavelet transform (DWT) and dimension reduction (DR), optimal band combination algorithm to five machine learning retrieval models using tree-based ensemble learning, neural network-based, and kernel-based. The comparison between the retrievals and Ni measurements shows that the DWT combined with t-distributed stochastic neighbor embedding (tSNE) coupled extreme gradient boosting (XGboost) retrieval model exhibited the best prediction for the validation dataset. Moreover, due to the integration of six statistical indicators of model performance and the fitted slope of the regression line, the retrieval framework can produce more robust and accurate predictions than those that rely on correlation coefficients. The demonstrated potential of spaceborne hyperspectral remote sensing to provide accurate quantitative measurements of soil heavy metal concentrations will serve as a reference for agricultural plot applications worldwide.

EMBEDR: Distinguishing signal from noise in single-cell omics data

Single-cell “omics”-based measurements are often high dimensional so that dimensionality reduction (DR) algorithms are necessary for data visualization and analysis. The lack of methods for separating signal from noise in DR outputs has limited their utility in generating data-driven discoveries in single-cell data. In this work we present EMBEDR, which assesses the output of any DR algorithm to distinguish evidence of structure from algorithm-induced noise in DR outputs. We apply EMBEDR to DR-generated representations of single-cell omics data of several modalities to show where they visually show real—not spurious—structure. EMBEDR generates a “p” value for each sample, allowing for direct comparisons of DR algorithms and facilitating optimization of algorithm hyperparameters. We show that the scale of a sample’s neighborhood can thus be determined and used to generate a novel “cell-wise optimal” embedding. EMBEDR is available as a Python package for immediate use.

An overview of the benefits and difficulties of dimensionality reduction

A novel algorithm for quantifying and identifying quality within embeddings of data

Quality can be optimized to find data scales and set algorithm parameters

A cell-wise view of quality generates robust and interpretable representations of data

Modern technologies have enabled biologists to construct enormous datasets containing millions of observations of thousands of measurements. These datasets push the limits of traditional analysis techniques, leaving doubts about the quality and fidelity of these methods. In this work, we present a sort of meta-algorithm, called EMBEDR, which seeks to evaluate when a certain class of methods, known as dimensionality reduction methods, are generating high-quality representations of data. We show that EMBEDR allows for visualizations of even large datasets to be interpreted with confidence. Furthermore, we show how asking about the method quality itself can lead to improved analyses of data. These improved analyses may directly impact our understanding of cellular biology, including how cells behave, grow, and respond to stimuli.

Tuning Database-Friendly Random Projection Matrices for Improved Distance Preservation on Specific Data

Random Projection is one of the most popular and successful dimensionality reduction algorithms for large volumes of data. However, given its stochastic nature, different initializations of the projection matrix can lead to very different levels of performance. This paper presents a guided random search algorithm to mitigate this problem. The proposed method uses a small number of training data samples to iteratively adjust a projection matrix, improving its performance on similarly distributed data. Experimental results show that projection matrices generated with the proposed method result in a better preservation of distances between data samples. Conveniently, this is achieved while preserving the database-friendliness of the projection matrix, as it remains sparse and comprised exclusively of integers after being tuned with our algorithm. Moreover, running the proposed algorithm on a consumer-grade CPU requires only a few seconds.