Zero-Shot Learning via Structure-Aligned Generative Adversarial Network

ZS-VAT: Learning Unbiased Attribute Knowledge for Zero-Shot Recognition Through Visual Attribute Transformer

In zero-shot learning (ZSL), attribute knowledge plays a vital role in transferring knowledge from seen classes to unseen classes. However, most existing ZSL methods learn biased attribute knowledge, which usually results in biased attribute prediction and a decline in zero-shot recognition performance. To solve this problem and learn unbiased attribute knowledge, we propose a visual attribute Transformer for zero-shot recognition (ZS-VAT), which is an effective and interpretable Transformer designed specifically for ZSL. In ZS-VAT, we design an attribute-head self-attention (AHSA) that is capable of learning unbiased attribute knowledge. Specifically, each attribute head in AHSA first transforms the local features into attribute-reinforced features and then accumulates the attribute knowledge from all corresponding reinforced features, reducing the mutual influence between attributes and avoiding information loss. AHSA finally preserves unbiased attribute knowledge through attribute embeddings. We also propose an attribute fusion model (AFM) that learns to recover the correct category knowledge from the attribute knowledge. In particular, AFM takes all features from AHSA as input and generates global embeddings. We carried out experiments to demonstrate that the attribute knowledge from AHSA and the category knowledge from AFM are able to assist each other. During the final semantic prediction, we combine the attribute embedding prediction (AEP) and global embedding prediction (GEP). We evaluated the proposed scheme on three benchmark datasets. ZS-VAT outperformed the state-of-the-art generalized ZSL (GZSL) methods on two datasets and achieved competitive results on the other dataset.

Generalized zero-shot learning via discriminative and transferable disentangled representations

In generalized zero-shot learning (GZSL), it is required to identify seen and unseen samples under the condition that only seen classes can be obtained during training. Recent methods utilize disentanglement to make the information contained in visual features semantically related, and ensuring semantic consistency and independence of the disentangled representations is the key to achieving better performance. However, we think there are still some limitations. Firstly, due to the fact that only seen classes can be obtained during training, the recognition of unseen samples will be poor. Secondly, the distribution relations of the representation space and the semantic space are different, and ignoring the discrepancy between them may impact the generalization of the model. In addition, the instances are associated with each other, and considering the interactions between them can obtain more discriminative information, which should not be ignored. Thirdly, since the synthesized visual features may not match the corresponding semantic descriptions well, it will compromise the learning of semantic consistency. To overcome these challenges, we propose to learn discriminative and transferable disentangled representations (DTDR) for generalized zero-shot learning. Firstly, we exploit the estimated class similarities to supervise the relations between seen semantic-matched representations and unseen semantic descriptions, thereby gaining better insight into the unseen domain. Secondly, we use cosine similarities between semantic descriptions to constrain the similarities between semantic-matched representations, thereby facilitating the distribution relation of semantic-matched representation space to approximate the distribution relation of semantic space. And during the process, the instance-level correlation can be taken into account. Thirdly, we reconstruct the synthesized visual features into the corresponding semantic descriptions to better establish the associations between them. The experimental results on four datasets verify the effectiveness of our method.

Fine-Grained Feature Generation for Generalized Zero-Shot Video Classification

Generalized zero-shot video classification aims to train a classifier to classify videos including both seen and unseen classes. Since the unseen videos have no visual information during training, most existing methods rely on the generative adversarial networks to synthesize visual features for unseen classes through the class embedding of category names. However, most category names only describe the content of the video, ignoring other relational information. As a rich information carrier, videos include actions, performers, environments, etc., and the semantic description of the videos also express the events from different levels of actions. In order to use fully explore the video information, we propose a fine-grained feature generation model based on video category name and its corresponding description texts for generalized zero-shot video classification. To obtain comprehensive information, we first extract content information from coarse-grained semantic information (category names) and motion information from fine-grained semantic information (description texts) as the base for feature synthesis. Then, we subdivide motion into hierarchical constraints on the fine-grained correlation between event and action from the feature level. In addition, we propose a loss that can avoid the imbalance of positive and negative examples to constrain the consistency of features at each level. In order to prove the validity of our proposed framework, we perform extensive quantitative and qualitative evaluations on two challenging datasets: UCF101 and HMDB51, and obtain a positive gain for the task of generalized zero-shot video classification.