Genie: A new, fast, and outlier-resistant hierarchical clustering algorithm

Super Partition: fast, flexible, and interpretable large-scale data reduction in R

Motivation

As data sets increase in size and complexity with advancing technology, flexible and interpretable data reduction methods that quantify information preservation become increasingly important.

Results

Super Partition is a large-scale approximation of the original Partition data reduction algorithm that allows the user to flexibly specify the minimum amount of information captured for each input feature. In an initial step, Genie, a fast, hierarchical clustering algorithm, forms a super-partition, thereby increasing the computational tractability by allowing Partition to be applied to the subsets. Applications to high dimensional data sets show scalability to hundreds of thousands of features with reasonable computation times.

Availability and implementation

Super Partition is a new function within the partition R package, available on the CRAN repository (https://cran.r-project.org/web/packages/partition/index.html).

Fast clustering algorithm based on MST of representative points

Minimum spanning tree (MST)-based clustering algorithms are widely used to detect clusters with diverse densities and irregular shapes. However, most algorithms require the entire dataset to construct an MST, which leads to significant computational overhead. To alleviate this issue, our proposed algorithm R-MST utilizes representative points instead of all sample points for constructing MST. Additionally, based on the density and nearest neighbor distance, we improved the representative point selection strategy to enhance the uniform distribution of representative points in sparse areas, enabling the algorithm to perform well on datasets with varying densities. Furthermore, traditional methods for eliminating inconsistent edges generally require prior knowledge about the number of clusters, which is not always readily available in practical applications. Therefore, we propose an adaptive method that employs mutual neighbors to identify inconsistent edges and determine the optimal number of clusters automatically. The experimental results indicate that the R-MST algorithm not only improves the efficiency of clustering but also enhances its accuracy.

Inferring population structure in biobank-scale genomic data

Inferring the structure of human populations from genetic variation data is a key task in population and medical genomic studies. Although a number of methods for population structure inference have been proposed, current methods are impractical to run on biobank-scale genomic datasets containing millions of individuals and genetic variants. We introduce SCOPE, a method for population structure inference that is orders of magnitude faster than existing methods while achieving comparable accuracy. SCOPE infers population structure in about a day on a dataset containing one million individuals and variants as well as on the UK Biobank dataset containing 488,363 individuals and 569,346 variants. Furthermore, SCOPE can leverage allele frequencies from previous studies to improve the interpretability of population structure estimates.

Visualization, benchmarking and characterization of nested single-cell heterogeneity as dynamic forest mixtures

A major topic of debate in developmental biology centers on whether development is continuous, discontinuous, or a mixture of both. Pseudo-time trajectory models, optimal for visualizing cellular progression, model cell transitions as continuous state manifolds and do not explicitly model real-time, complex, heterogeneous systems and are challenging for benchmarking with temporal models. We present a data-driven framework that addresses these limitations with temporal single-cell data collected at discrete time points as inputs and a mixture of dependent minimum spanning trees (MSTs) as outputs, denoted as dynamic spanning forest mixtures (DSFMix). DSFMix uses decision-tree models to select genes that account for variations in multimodality, skewness and time. The genes are subsequently used to build the forest using tree agglomerative hierarchical clustering and dynamic branch cutting. We first motivate the use of forest-based algorithms compared to single-tree approaches for visualizing and characterizing developmental processes. We next benchmark DSFMix to pseudo-time and temporal approaches in terms of feature selection, time correlation, and network similarity. Finally, we demonstrate how DSFMix can be used to visualize, compare and characterize complex relationships during biological processes such as epithelial-mesenchymal transition, spermatogenesis, stem cell pluripotency, early transcriptional response from hormones and immune response to coronavirus disease. Our results indicate that the expression of genes during normal development exhibits a high proportion of non-uniformly distributed profiles that are mostly right-skewed and multimodal; the latter being a characteristic of major steady states during development. Our study also identifies and validates gene signatures driving complex dynamic processes during somatic or germline differentiation.

WEClustering: word embeddings based text clustering technique for large datasets

A massive amount of textual data now exists in digital repositories in the form of research articles, news articles, reviews, Wikipedia articles, and books, etc. Text clustering is a fundamental data mining technique to perform categorization, topic extraction, and information retrieval. Textual datasets, especially which contain a large number of documents are sparse and have high dimensionality. Hence, traditional clustering techniques such as K-means, Agglomerative clustering, and DBSCAN cannot perform well. In this paper, a clustering technique especially suitable to large text datasets is proposed that overcome these limitations. The proposed technique is based on word embeddings derived from a recent deep learning model named “Bidirectional Encoders Representations using Transformers”. The proposed technique is named as WEClustering. The proposed technique deals with the problem of high dimensionality in an effective manner, hence, more accurate clusters are formed. The technique is validated on several datasets of varying sizes and its performance is compared with other widely used and state of the art clustering techniques. The experimental comparison shows that the proposed clustering technique gives a significant improvement over other techniques as measured by metrics such Purity and Adjusted Rand Index.

A Hybrid Model to Classify Patients with Chronic Obstructive Respiratory Diseases

Over the last decades, an increase in the ageing population and age-related diseases has been observed, with the increase in healthcare costs. As so, new solutions to provide more efficient and affordable support to this group of patients are needed. Such solutions should never discard the user and instead should focus on promoting more healthy lifestyles and provide tools for patients’ active participation in the treatment and management of their diseases. In this concern, the Personal Health Empowerment (PHE) project presented in this paper aims to empower patients to monitor and improve their health, using personal data and technology assisted coaching. The work described in this paper focuses on defining an approach for user modelling on patients with chronic obstructive respiratory diseases using a hybrid modelling approach to identify different groups of users. A classification model with 90.4% prediction accuracy was generated combining agglomerative hierarchical clustering and decision tree classification techniques. Furthermore, this model identified 5 clusters which describe characteristics of 5 different types of users according to 7 generated rules. With the modelling approach defined in this study, a personalized coaching solution will be built considering patients with different necessities and capabilities and adapting the support provided, enabling the recognition of early signs of exacerbations and objective self-monitoring and treatment of the disease. The novel factor of this approach resides in the possibility to integrate personalized coaching technologies adapted to each kind of user within a smartphone-based application resulting in a reliable and affordable alternative for patients to manage their disease.

Weighted z-Distance-Based Clustering and Its Application to Time-Series Data

Clustering is the practice of dividing given data into similar groups and is one of the most widely used methods for unsupervised learning. Lee and Ouyang proposed a self-constructing clustering (SCC) method in which the similarity threshold, instead of the number of clusters, is specified in advance by the user. For a given set of instances, SCC performs only one training cycle on those instances. Once an instance has been assigned to a cluster, the assignment will not be changed afterwards. The clusters produced may depend on the order in which the instances are considered, and assignment errors are more likely to occur. Also, all dimensions are equally weighted, which may not be suitable in certain applications, e.g., time-series clustering. In this paper, improvements are proposed. Two or more training cycles on the instances are performed. An instance can be re-assigned to another cluster in each cycle. In this way, the clusters produced are less likely to be affected by the feeding order of the instances. Also, each dimension of the input can be weighted differently in the clustering process. The values of the weights are adaptively learned from the data. A number of experiments with real-world benchmark datasets are conducted and the results are shown to demonstrate the effectiveness of the proposed ideas.

Antibiotic resistance genes are abundant and diverse in raw sewage used for urban agriculture in Africa and associated with urban population density

A comparative study was conducted to (1) assess the potential of raw sewage used for urban agriculture to disseminate bacterial resistance in two cities of different size in Cameroon (Central Africa) and (2) compare the outcome with data obtained in Burkina Faso (West Africa). In each city, raw sewage samples were sampled from open-air canals in three neighbourhoods. After DNA extraction, the microbial population structure and function, presence of pathogens, antibiotic resistance genes and Enterobacteriaceae plasmids replicons were analysed using whole genome shotgun sequencing and bioinformatics. Forty-three pathogen-specific virulenc e factor genes were detected in the sewage. Eighteen different incompatibility groups of Enterobacteriaceae plasmid replicon types (ColE, A/C, B/O/K/Z, FIA, FIB, FIC, FII, H, I, N, P, Q, R, T, U, W, X, and Y) implicated in the spread of drug-resistance genes were present in the sewage samples. One hundred thirty-six antibiotic resistance genes commonly associated with MDR plasmid carriage were identified in both cities. Enterobacteriaceae plasmid replicons and ARGs found in Burkina Faso wastewaters were also present in Cameroon waters. The abundance of Enterobacteriaceae, plasmid replicons and antibiotic resistance genes was greater in Yaounde, the city with the greater population. In conclusion, the clinically relevant environmental resistome found in raw sewage used for urban agriculture is common in West and Central Africa. The size of the city impacts on the abundance of drug-resistant genes in the raw sewage while ESBL gene abundance is related to the prevalence of Enterobacteriaceae along with plasmid Enterobacteriaceae abundance associated to faecal pollution.

Fast hierarchical Bayesian analysis of population structure

We present fastbaps, a fast solution to the genetic clustering problem. Fastbaps rapidly identifies an approximate fit to a Dirichlet process mixture model (DPM) for clustering multilocus genotype data. Our efficient model-based clustering approach is able to cluster datasets 10-100 times larger than the existing model-based methods, which we demonstrate by analyzing an alignment of over 110 000 sequences of HIV-1 pol genes. We also provide a method for rapidly partitioning an existing hierarchy in order to maximize the DPM model marginal likelihood, allowing us to split phylogenetic trees into clades and subclades using a population genomic model. Extensive tests on simulated data as well as a diverse set of real bacterial and viral datasets show that fastbaps provides comparable or improved solutions to previous model-based methods, while being significantly faster. The method is made freely available under an open source MIT licence as an easy to use R package at https://github.com/gtonkinhill/fastbaps.

Comparing the qualitative performances of handheld NIR and Raman spectrophotometers for the detection of falsified pharmaceutical products

Over the last decade, the growth of the global pharmaceutical market has led to an overall increase of substandard and falsified drugs especially on the African market (or emerging countries). Recently, several methods using handheld/portable vibrational spectroscopy have been developed for rapid and on-field drug analysis. The objective of this work was to evaluate the performances of various NIR and Raman handheld spectrophotometers in specific brand identification of medicines through their primary packaging. Three groups of drug samples (artemether-lumefantrine, paracetamol and ibuprofen) were used in tablet or capsule forms. In order to perform a critical comparison, the analytical performances of the two analytical systems were compared statistically using three methods: hierarchical clustering algorithm (HCA), data-driven soft independent modelling of class analogy (DD-SIMCA) and hit quality index (HQI). The overall results show good detection abilities for NIR systems compared to Raman systems based on Matthews's correlation coefficients, generally close to one. Raman systems are less sensitive to the physical state of the samples than the NIR systems, it also suffers of the auto-fluorescence phenomenon and the signal of highly dosed active pharmaceutical ingredient (e.g. paracetamol or lumefantrine) may mask the signal of low-dosed and weaker Raman active compounds (e.g. artemether). Hence, Raman systems are less effective for specific product identification purposes but are interesting in the context of falsification because they allow a visual interpretation of the spectral signature (presence or absence of API).

Penalty-Based and Other Representations of Economic Inequality

Economic inequality measures are employed as a key component in various socio-demographic indices to capture the disparity between the wealthy and poor. Since their inception, they have also been used as a basis for modelling spread and disparity in other contexts. While recent research has identified that a number of classical inequality and welfare functions can be considered in the framework of OWA operators, here we propose a framework of penalty-based aggregation functions and their associated penalties as measures of inequality.

A modified two-stage Markov clustering algorithm for large and sparse networks

BACKGROUND

Graph-based hierarchical clustering algorithms become prohibitively costly in both execution time and storage space, as the number of nodes approaches the order of millions.

OBJECTIVE

A fast and highly memory efficient Markov clustering algorithm is proposed to perform the classification of huge sparse networks using an ordinary personal computer.

METHODS

Improvements compared to previous versions are achieved through adequately chosen data structures that facilitate the efficient handling of symmetric sparse matrices. Clustering is performed in two stages: the initial connected network is processed in a sparse matrix until it breaks into isolated, small, and relatively dense subgraphs, which are then processed separately until convergence is obtained. An intelligent stopping criterion is also proposed to quit further processing of a subgraph that tends toward completeness with equal edge weights. The main advantage of this algorithm is that the necessary number of iterations is separately decided for each graph node.

RESULTS

The proposed algorithm was tested using the SCOP95 and large synthetic protein sequence data sets. The validation process revealed that the proposed method can reduce 3-6 times the processing time of huge sequence networks compared to previous Markov clustering solutions, without losing anything from the partition quality.

CONCLUSIONS

A one-million-node and one-billion-edge protein sequence network defined by a BLAST similarity matrix can be processed with an upper-class personal computer in 100 minutes. Further improvement in speed is possible via parallel data processing, while the extension toward several million nodes needs intermediary data storage, for example on solid state drives.