Probabilistic Semi-Supervised Learning via Sparse Graph Structure Learning

Taming Overconfident Prediction on Unlabeled Data From Hindsight

Minimizing prediction uncertainty on unlabeled data is a key factor to achieve good performance in semi-supervised learning (SSL). The prediction uncertainty is typically expressed as the entropy computed by the transformed probabilities in output space. Most existing works distill low-entropy prediction by either accepting the determining class (with the largest probability) as the true label or suppressing subtle predictions (with the smaller probabilities). Unarguably, these distillation strategies are usually heuristic and less informative for model training. From this discernment, this article proposes a dual mechanism, named adaptive sharpening (ADS), which first applies a soft-threshold to adaptively mask out determinate and negligible predictions, and then seamlessly sharpens the informed predictions, distilling certain predictions with the informed ones only. More importantly, we theoretically analyze the traits of ADS by comparing it with various distillation strategies. Numerous experiments verify that ADS significantly improves state-of-the-art SSL methods by making it a plug-in. Our proposed ADS forges a cornerstone for future distillation-based SSL research.

Homophily-Enhanced Self-Supervision for Graph Structure Learning: Insights and Directions

Graph neural networks (GNNs) have recently achieved remarkable success on a variety of graph-related tasks, while such success relies heavily on a given graph structure that may not always be available in real-world applications. To address this problem, graph structure learning (GSL) is emerging as a promising research topic where task-specific graph structure and GNN parameters are jointly learned in an end-to-end unified framework. Despite their great progress, existing approaches mostly focus on the design of similarity metrics or graph construction, but directly default to adopting downstream objectives as supervision, which lacks deep insight into the power of supervision signals. More importantly, these approaches struggle to explain how GSL helps GNNs, and when and why this help fails. In this article, we conduct a systematic experimental evaluation to reveal that GSL and GNNs enjoy consistent optimization goals in terms of improving the graph homophily. Furthermore, we demonstrate theoretically and experimentally that task-specific downstream supervision may be insufficient to support the learning of both graph structure and GNN parameters, especially when the labeled data are extremely limited. Therefore, as a complement to downstream supervision, we propose homophily-enhanced self-supervision for GSL (HES-GSL), a method that provides more supervision for learning an underlying graph structure. A comprehensive experimental study demonstrates that HES-GSL scales well to various datasets and outperforms other leading methods. Our code will be available in https://github.com/LirongWu/Homophily-Enhanced-Self-supervision.

GUIDE: Training Deep Graph Neural Networks via Guided Dropout Over Edges

Graph neural networks (GNNs) have made great progress in graph-based semi-supervised learning (GSSL). However, most existing GNNs are confronted with the oversmoothing issue that limits their expressive ability. A key factor that leads to this problem is the excessive aggregation of information from other classes when updating the node representation. To alleviate this limitation, we propose an effective method called GUIded Dropout over Edges (GUIDE) for training deep GNNs. The core of the method is to reduce the influence of nodes from other classes by removing a certain number of inter-class edges. In GUIDE, we drop edges according to the edge strength, which is defined as the time an edge acts as a bridge along the shortest path between node pairs. We find that the stronger the edge strength, the more likely it is to be an inter-class edge. In this way, GUIDE can drop more inter-class edges and keep more intra-class edges. Therefore, nodes in the same community or class are more similar, whereas different classes are more separated in the embedded space. In addition, we perform some theoretical analysis of the proposed method, which explains why it is effective in alleviating the oversmoothing problem. To validate its rationality and effectiveness, we conduct experiments on six public benchmarks with different GNNs backbones. Experimental results demonstrate that GUIDE consistently outperforms state-of-the-art methods in both shallow and deep GNNs.