Orthogonal Nonnegative Matrix Factorization using a novel deep Autoencoder Network

Interpretable unsupervised neural network structure for data clustering via differentiable reconstruction of ONMF and sparse autoencoder

Neural networks, while powerful, often face significant challenges in terms of interpretability, particularly in clustering tasks. Traditional methods typically rely on post-hoc explanations or supervised learning, which limit their ability to provide transparent, understandable results. The innovation of this method is the integration of Orthogonal Non-negative Matrix Factorization (ONMF) into a clustering layer, which is directly incorporated into the OSINN neural network model and enhanced by a Sparse Autoencoder (SAE). This integration allows for end-to-end model training, which improves clustering performance and provides interpretability by differentiably reconstructing ONMF and extracting sparse features with SAE. OSINN's main innovation is the differentiable reconstruction of ONMF, which creates a transparent clustering layer that is both end-to-end trainable and easily interpretable. Firstly, OSINN, as a transparent neural network with an embedded clustering layer, allows for pre-training model decision predictions, where the model is first trained on a large set of unlabeled data to learn useful features, and then makes clustering decisions based on these learned features. Secondly, by using unsupervised neural networks for clustering, OSINN can handle more complex data, where the network automatically groups similar data points without needing labeled examples, which is especially useful for tasks involving large, unlabeled datasets. Thirdly, as a differentiable version of ONMF, OSINN excels at data clustering by transforming the ONMF method into a differentiable process, making it easy to integrate into the neural network architecture for end-to-end learning, which in turn enhances both interpretability and performance. Experimental results on MNIST, CIFAR-10, Fashion-MNIST, and CIFAR-100 datasets show that OSINN outperforms existing methods, achieving clustering accuracies of 90%, 24%, 64%, and 44%, respectively. Compared to traditional clustering algorithms, OSINN improves accuracy by over 10%, outperforms deep clustering methods by more than 1.2%, and surpasses Non-negative Matrix Factorization (NMF) and Autoencoder (AE) variants by over 1.5%.

Comprehensive Multiview Representation Learning via Deep Autoencoder-Like Nonnegative Matrix Factorization

Learning a comprehensive representation from multiview data is crucial in many real-world applications. Multiview representation learning (MRL) based on nonnegative matrix factorization (NMF) has been widely adopted by projecting high-dimensional space into a lower order dimensional space with great interpretability. However, most prior NMF-based MRL techniques are shallow models that ignore hierarchical information. Although deep matrix factorization (DMF)-based methods have been proposed recently, most of them only focus on the consistency of multiple views and have cumbersome clustering steps. To address the above issues, in this article, we propose a novel model termed deep autoencoder-like NMF for MRL (DANMF-MRL), which obtains the representation matrix through the deep encoding stage and decodes it back to the original data. In this way, through a DANMF-based framework, we can simultaneously consider the multiview consistency and complementarity, allowing for a more comprehensive representation. We further propose a one-step DANMF-MRL, which learns the latent representation and final clustering labels matrix in a unified framework. In this approach, the two steps can negotiate with each other to fully exploit the latent clustering structure, avoid previous tedious clustering steps, and achieve optimal clustering performance. Furthermore, two efficient iterative optimization algorithms are developed to solve the proposed models both with theoretical convergence analysis. Extensive experiments on five benchmark datasets demonstrate the superiority of our approaches against other state-of-the-art MRL methods.

Difference of Convex Functions Programming With Machine-Learning Prior for the Imaging Problem in Electrical Capacitance Tomography

The electrical capacitance tomography technology has potential benefits for the process industry by providing visualization of material distributions. One of the main technical gaps and impediments that must be overcome is the low-quality tomogram. To address this problem, this study introduces the data-guided prior and combines it with the electrical measurement mechanism and the sparsity prior to produce a new difference of convex functions programming problem that turns the image reconstruction problem into an optimization problem. The data-guided prior is learned from a provided dataset and captures the details of imaging targets since it is a specific image. A new numerical scheme that allows a complex optimization problem to be split into a few less difficult subproblems is developed to solve the challenging difference of convex functions programming problem. A new dimensionality reduction method is developed and combined with the relevance vector machine to generate a new learning engine for the forecast of the data-guided prior. The new imaging method fuses multisource information and unifies data-guided and measurement physics modeling paradigms. Performance evaluation results have validated that the new method successfully works on a series of test tasks with higher reconstruction quality and lower noise sensitivity than the popular imaging methods.

MDSR-NMF: Multiple deconstruction single reconstruction deep neural network model for non-negative matrix factorization

Dimension reduction is one of the most sought-after strategies to cope with high-dimensional ever-expanding datasets. To address this, a novel deep-learning architecture has been designed with multiple deconstruction and single reconstruction layers for non-negative matrix factorization aimed at low-rank approximation. This design ensures that the reconstructed input matrix has a unique pair of factor matrices. The two-stage approach, namely, pretraining and stacking, aids in the robustness of the architecture. The sigmoid function has been adjusted in such a way that fulfils the non-negativity criteria and also helps to alleviate the data-loss problem. Xavier initialization technique aids in the solution of the exploding or vanishing gradient problem. The objective function involves regularizer that ensures the best possible approximation of the input matrix. The superior performance of MDSR-NMF, over six well-known dimension reduction methods, has been demonstrated extensively using five datasets for classification and clustering. Computational complexity and convergence analysis have also been presented to establish the model.

Rethinking adversarial domain adaptation: Orthogonal decomposition for unsupervised domain adaptation in medical image segmentation

Medical image segmentation methods based on deep learning have made remarkable progress. However, such existing methods are sensitive to data distribution. Therefore, slight domain shifts will cause a decline of performance in practical applications. To relieve this problem, many domain adaptation methods learn domain-invariant representations by alignment or adversarial training whereas ignoring domain-specific representations. In response to this issue, this paper rethinks the traditional domain adaptation framework and proposes a novel orthogonal decomposition adversarial domain adaptation (ODADA) architecture for medical image segmentation. The main idea behind our proposed ODADA model is to decompose the input features into domain-invariant and domain-specific representations and then use the newly designed orthogonal loss function to encourage their independence. Furthermore, we propose a two-step optimization strategy to extract domain-invariant representations by separating domain-specific representations, fighting the performance degradation caused by domain shifts. Encouragingly, the proposed ODADA framework is plug-and-play and can replace the traditional adversarial domain adaptation module. The proposed method has consistently demonstrated effectiveness through comprehensive experiments on three publicly available datasets, including cross-site prostate segmentation dataset, cross-site COVID-19 lesion segmentation dataset, and cross-modality cardiac segmentation dataset. The source code is available at https://github.com/YonghengSun1997/ODADA.