Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

Unsupervised Dual Deep Hashing With Semantic-Index and Content-Code for Cross-Modal Retrieval

Hashing technology has exhibited great cross-modal retrieval potential due to its appealing retrieval efficiency and storage effectiveness. Most current supervised cross-modal retrieval methods heavily rely on accurate semantic supervision, which is intractable for annotations with ever-growing sample sizes. By comparison, the existing unsupervised methods rely on accurate sample similarity preservation strategies with intensive computational costs to compensate for the lack of semantic guidance, which causes these methods to lose the power to bridge the semantic gap. Furthermore, both kinds of approaches need to search for the nearest samples among all samples in a large search space, whose process is laborious. To address these issues, this paper proposes an unsupervised dual deep hashing (UDDH) method with semantic-index and content-code for cross-modal retrieval. Deep hashing networks are utilized to extract deep features and jointly encode the dual hashing codes in a collaborative manner with a common semantic index and modality content codes to simultaneously bridge the semantic and heterogeneous gaps for cross-modal retrieval. The dual deep hashing architecture, comprising the head code on semantic index and tail codes on modality content, enhances the efficiency for cross-modal retrieval. A query sample only needs to search for the retrieved samples with the same semantic index, thus greatly shrinking the search space and achieving superior retrieval efficiency. UDDH integrates the learning processes of deep feature extraction, binary optimization, common semantic index, and modality content code within a unified model, allowing for collaborative optimization to enhance the overall performance. Extensive experiments are conducted to demonstrate the retrieval superiority of the proposed approach over the state-of-the-art baselines.

A novel multi-feature learning model for disease diagnosis using face skin images

BACKGROUND

Facial skin characteristics can provide valuable information about a patient's underlying health conditions.

OBJECTIVE

In practice, there are often samples with divergent characteristics (commonly known as divergent samples) that can be attributed to environmental factors, living conditions, or genetic elements. These divergent samples significantly degrade the accuracy of diagnoses.

METHODOLOGY

To tackle this problem, we propose a novel multi-feature learning method called Multi-Feature Learning with Centroid Matrix (MFLCM), which aims to mitigate the influence of divergent samples on the accurate classification of samples located on the boundary. In this approach, we introduce a novel discriminator that incorporates a centroid matrix strategy and simultaneously adapt it to a classifier in a unified model. We effectively apply the centroid matrix to the embedding feature spaces, which are transformed from the multi-feature observation space, by calculating a relaxed Hamming distance. The purpose of the centroid vectors for each category is to act as anchors, ensuring that samples from the same class are positioned close to their corresponding centroid vector while being pushed further away from the remaining centroids.

RESULTS

Validation of the proposed method with clinical facial skin dataset showed that the proposed method achieved F1 scores of 92.59%, 83.35%, 82.84% and 85.46%, respectively for the detection the Healthy, Diabetes Mellitus (DM), Fatty Liver (FL) and Chronic Renal Failure (CRF).

CONCLUSION

Experimental results demonstrate the superiority of the proposed method compared with typical classifiers single-view-based and state-of-the-art multi-feature approaches. To the best of our knowledge, this study represents the first to demonstrate concept of multi-feature learning using only facial skin images as an effective non-invasive approach for simultaneously identifying DM, FL and CRF in Han Chinese, the largest ethnic group in the world.

Harmonization Shared Autoencoder Gaussian Process Latent Variable Model With Relaxed Hamming Distance

Multiview learning has shown its superiority in visual classification compared with the single-view-based methods. Especially, due to the powerful representation capacity, the Gaussian process latent variable model (GPLVM)-based multiview approaches have achieved outstanding performances. However, most of them only follow the assumption that the shared latent variables can be generated from or projected to the multiple observations but fail to exploit the harmonization in the back constraint and adaptively learn a classifier according to these learned variables, which would result in performance degradation. To tackle these two issues, in this article, we propose a novel harmonization shared autoencoder GPLVM with a relaxed Hamming distance (HSAGP-RHD). Particularly, an autoencoder structure with the Gaussian process (GP) prior is first constructed to learn the shared latent variable for multiple views. To enforce the agreement among various views in the encoder, a harmonization constraint is embedded into the model by making consistency for the view-specific similarity. Furthermore, we also propose a novel discriminative prior, which is directly imposed on the latent variable to simultaneously learn the fused features and adaptive classifier in a unit model. In detail, the centroid matrix corresponding to the centroids of different categories is first obtained. A relaxed Hamming distance (RHD)-based measurement is subsequently presented to measure the similarity and dissimilarity between the latent variable and centroids, not only allowing us to get the closed-form solutions but also encouraging the points belonging to the same class to be close, while those belonging to different classes to be far. Due to this novel prior, the category of the out-of-sample is also allowed to be simply assigned in the testing phase. Experimental results conducted on three real-world data sets demonstrate the effectiveness of the proposed method compared with state-of-the-art approaches.

Gaussian process linking functions for mind, brain, and behavior

The link between mind, brain, and behavior has mystified philosophers and scientists for millennia. Recent progress has been made by forming statistical associations between manifest variables of the brain (e.g., electroencephalogram [EEG], functional MRI [fMRI]) and manifest variables of behavior (e.g., response times, accuracy) through hierarchical latent variable models. Within this framework, one can make inferences about the mind in a statistically principled way, such that complex patterns of brain-behavior associations drive the inference procedure. However, previous approaches were limited in the flexibility of the linking function, which has proved prohibitive for understanding the complex dynamics exhibited by the brain. In this article, we propose a data-driven, nonparametric approach that allows complex linking functions to emerge from fitting a hierarchical latent representation of the mind to multivariate, multimodal data. Furthermore, to enforce biological plausibility, we impose both spatial and temporal structure so that the types of realizable system dynamics are constrained. To illustrate the benefits of our approach, we investigate the model's performance in a simulation study and apply it to experimental data. In the simulation study, we verify that the model can be accurately fitted to simulated data, and latent dynamics can be well recovered. In an experimental application, we simultaneously fit the model to fMRI and behavioral data from a continuous motion tracking task. We show that the model accurately recovers both neural and behavioral data and reveals interesting latent cognitive dynamics, the topology of which can be contrasted with several aspects of the experiment.