Unsupervised graph-level representation learning with hierarchical contrasts

Implementing the discontinuous-Galerkin finite element method using graph neural networks with application to diffusion equations

Machine learning (ML) has benefited from both software and hardware advancements, leading to increasing interest in capitalising on ML throughout academia and industry. There have been efforts in the scientific computing community to leverage this development via implementing conventional partial differential equation (PDE) solvers with machine learning packages, most of which rely on structured spatial discretisation and fast convolution algorithms. However, unstructured meshes are favoured in problems with complex geometries. To bridge this gap, we propose to implement the unstructured Finite Element Method (FEM) on simplicial meshes with graph neural networks. This paper is the first to implement an unstructured mesh FEM solver using graph neural networks. All compute-intensive algorithms in the solver are represented with either convolutional or graph neural networks. Specifically, the FEM solver uses a discontinuous Galerkin formulation with an interior penalty method for spatial discretisation and a multigrid preconditioned Krylov solver as the linear solver. The multigrid method has been designed to suit the data structure within the ML package and adopts the commonly used U-Net architecture for this. A hierarchy of coarsened meshes is generated from p-multigrid and algebraic node agglomeration guided by either a space-filling curve or a smoothed aggregation algorithm. The solver is verified and assessed for solving the diffusion problems. The solver shows the theoretical convergence rate of (p+1) order. Compared with a highly optimised implementation, the solver running on GPU can reach promising throughput in terms of matrix operator evaluation at 6.8 MDOF/s. The method can easily extend to other PDEs and computing platforms beyond CPU and GPU.

Advancing Sports Cardiology: Integrating Artificial Intelligence with Wearable Devices for Cardiovascular Health Management

Sports cardiology focuses on athletes' cardiovascular health, yet sudden cardiac death remains a significant concern despite preventative measures. Prolonged physical activity leads to notable cardiovascular adaptations, known as the athlete's heart, which can resemble certain pathological conditions, complicating accurate diagnoses and potentially leading to serious consequences such as unnecessary exclusion from sports or missed treatment opportunities. Wearable devices, including smartwatches and smart glasses, have become prevalent for monitoring health metrics, offering potential clinical applications for sports cardiologists. These gadgets are capable of spotting exercise-induced arrhythmias, uncovering hidden heart problems, and offering crucial information for training and recovery, to minimize exercise-related cardiac incidents and enhance heart health care. However, concerns about data accuracy and the actionable value of the obtained information persist. A major challenge lies in the integration of artificial intelligence with wearables, research gaps remain regarding their ability to provide real-time, reliable, and clinically relevant insights. Combining artificial intelligence with wearable devices can improve how data is managed and used in sports cardiology. Artificial intelligence, particularly machine learning, can classify, predict, and draw inferences from the data collected by wearables, revolutionizing patient data usage. Despite artificial intelligence's proven effectiveness in managing chronic conditions, the limited research on its application in sports cardiology, particularly regarding wearables, creates a critical gap that needs to be addressed. This review examines commercially available wearables and their applications in sports cardiology, exploring how artificial intelligence can be integrated into wearable technology to advance the field.

Graph Batch Coarsening framework for scalable graph neural networks

Due to the neighborhood explosion phenomenon, scaling up graph neural networks to large graphs remains a huge challenge. Various sampling-based mini-batch approaches, such as node-wise, layer-wise, and subgraph sampling, have been proposed to alleviate this issue. However, intensive random sampling incurs additional overhead during training and often fails to deliver good performance consistently. To surmount these limitations, we propose Graph Batch Coarsening (GBC), a simple and general graph batching framework designed to facilitate scalable training of arbitrary GNN models. GBC preprocesses the input graph and generates a set of much smaller subgraphs to be used as mini-batches; then any GNN model can be trained only on those small graphs. This framework avoids random sampling completely and makes no extra change on the backbone GNN models including hyperparameters. To implement the framework, we present a graph decomposition method based on label propagation and a novel graph coarsening algorithm designed for training GNN. Empirically, GBC demonstrates superior performance in accuracy, training time and memory usage on various small to large-scale graphs.

GTC: GNN-Transformer co-contrastive learning for self-supervised heterogeneous graph representation

Graph Neural Networks (GNNs) have emerged as the most powerful weapon for various graph tasks due to the message-passing mechanism's great local information aggregation ability. However, over-smoothing has always hindered GNNs from going deeper and capturing multi-hop neighbors. Meanwhile, most methods follow a semi-supervised learning manner, the label scarcity would limit their applicability in real-world systems. Unlike GNNs, Transformers can model global information and multi-hop interactions via multi-head self-attention and a proper Transformer structure can show more immunity to over-smoothing. So, can we propose a novel framework to combine GNN and Transformer, integrating both GNN's local information aggregation and Transformer's global information modeling ability to eliminate the over-smoothing problem and achieve self-supervised graph representation? To realize this, this paper proposes a collaborative learning scheme for GNN-Transformer and constructs GTC architecture. GTC leverages the GNN and Transformer branch to encode node information from different views respectively, and establishes contrastive learning tasks based on the encoded cross-view information to realize self-supervised heterogeneous graph representation. For the Transformer branch, we propose Metapath-aware Hop2Token and CG-Hetphormer, which can cooperate with GNNs to attentively encode neighborhood information from different levels. As far as we know, this is the first attempt in the field of graph representation learning to utilize both GNNs and Transformer to collaboratively capture different view information and conduct cross-view contrastive learning. The experiments on real datasets show that GTC exhibits superior performance compared with state-of-the-art methods. Codes can be available at https://github.com/PHD-lanyu/GTC.

GMNI: Achieve good data augmentation in unsupervised graph contrastive learning

Graph contrastive learning (GCL) shows excellent potential in unsupervised graph representation learning. Data augmentation (DA), responsible for generating diverse views, plays a vital role in GCL, and its optimal choice heavily depends on the downstream task. However, it is impossible to measure task-relevant information under an unsupervised setting. Therefore, many GCL methods risk insufficient information by failing to preserve essential information necessary for the downstream task or risk encoding redundant information. In this paper, we propose a novel method called Minimal Noteworthy Information for unsupervised Graph contrastive learning (GMNI), featuring automated DA. It achieves good DA by balancing missing and excessive information, approximating the optimal views in contrastive learning. We employ an adversarial training strategy to generate views that share minimal noteworthy information (MNI), reducing nuisance information by minimization optimization and ensuring sufficient information by emphasizing noteworthy information. Besides, we introduce randomness based on MNI to augmentation, thereby enhancing view diversity and stabilizing the model against perturbations. Extensive experiments on unsupervised and semi-supervised learning over 14 datasets demonstrate the superiority of GMNI over GCL methods with automated and manual DA. GMNI achieves up to a 1.64% improvement over the state-of-the-art in unsupervised node classification, up to a 1.97% improvement in unsupervised graph classification, and up to a 3.57% improvement in semi-supervised graph classification.

Backdoor attacks on unsupervised graph representation learning

Unsupervised graph learning techniques have garnered increasing interest among researchers. These methods employ the technique of maximizing mutual information to generate representations of nodes and graphs. We show that these methods are susceptible to backdoor attacks, wherein the adversary can poison a small portion of unlabeled graph data (e.g., node features and graph structure) by introducing triggers into the graph. This tampering disrupts the representations and increases the risk to various downstream applications. Previous backdoor attacks in supervised learning primarily operate directly on the label space and may not be suitable for unlabeled graph data. To tackle this challenge, we introduce GRBA,1 a gradient-based first-order backdoor attack method. To the best of our knowledge, this constitutes a pioneering endeavor in investigating backdoor attacks within the domain of unsupervised graph learning. The initiation of this method does not necessitate prior knowledge of downstream tasks, as it directly focuses on representations. Furthermore, it is versatile and can be applied to various downstream tasks, including node classification, node clustering and graph classification. We evaluate GRBA on state-of-the-art unsupervised learning models, and the experimental results substantiate the effectiveness and evasiveness of GRBA in both node-level and graph-level tasks.

Harnessing collective structure knowledge in data augmentation for graph neural networks

Graph neural networks (GNNs) have achieved state-of-the-art performance in graph representation learning. Message passing neural networks, which learn representations through recursively aggregating information from each node and its neighbors, are among the most commonly-used GNNs. However, a wealth of structural information of individual nodes and full graphs is often ignored in such process, which restricts the expressive power of GNNs. Various graph data augmentation methods that enable the message passing with richer structure knowledge have been introduced as one main way to tackle this issue, but they are often focused on individual structure features and difficult to scale up with more structure features. In this work we propose a novel approach, namely collective structure knowledge-augmented graph neural network (CoS-GNN), in which a new message passing method is introduced to allow GNNs to harness a diverse set of node- and graph-level structure features, together with original node features/attributes, in augmented graphs. In doing so, our approach largely improves the structural knowledge modeling of GNNs in both node and graph levels, resulting in substantially improved graph representations. This is justified by extensive empirical results where CoS-GNN outperforms state-of-the-art models in various graph-level learning tasks, including graph classification, anomaly detection, and out-of-distribution generalization.

Redundancy-Free Self-Supervised Relational Learning for Graph Clustering

Graph clustering, which learns the node representations for effective cluster assignments, is a fundamental yet challenging task in data analysis and has received considerable attention accompanied by graph neural networks (GNNs) in recent years. However, most existing methods overlook the inherent relational information among the nonindependent and nonidentically distributed nodes in a graph. Due to the lack of exploration of relational attributes, the semantic information of the graph-structured data fails to be fully exploited which leads to poor clustering performance. In this article, we propose a novel self-supervised deep graph clustering method named relational redundancy-free graph clustering (R2FGC) to tackle the problem. It extracts the attribute- and structure-level relational information from both global and local views based on an autoencoder (AE) and a graph AE (GAE). To obtain effective representations of the semantic information, we preserve the consistent relationship among augmented nodes, whereas the redundant relationship is further reduced for learning discriminative embeddings. In addition, a simple yet valid strategy is used to alleviate the oversmoothing issue. Extensive experiments are performed on widely used benchmark datasets to validate the superiority of our R2FGC over state-of-the-art baselines. Our codes are available at https://github.com/yisiyu95/R2FGC.

Artificial intelligence in pathologic diagnosis, prognosis and prediction of prostate cancer

Histopathology, which is the gold-standard for prostate cancer diagnosis, faces significant challenges. With prostate cancer ranking among the most common cancers in the United States and worldwide, pathologists experience an increased number for prostate biopsies. At the same time, precise pathological assessment and classification are necessary for risk stratification and treatment decisions in prostate cancer care, adding to the challenge to pathologists. Recent advancement in digital pathology makes artificial intelligence and learning tools adopted in histopathology feasible. In this review, we introduce the concept of AI and its various techniques in the field of histopathology. We summarize the clinical applications of AI pathology for prostate cancer, including pathological diagnosis, grading, prognosis evaluation, and treatment options. We also discuss how AI applications can be integrated into the routine pathology workflow. With these rapid advancements, it is evident that AI applications in prostate cancer go beyond the initial goal of being tools for diagnosis and grading. Instead, pathologists can provide additional information to improve long-term patient outcomes by assessing detailed histopathologic features at pixel level using digital pathology and AI. Our review not only provides a comprehensive summary of the existing research but also offers insights for future advancements.

A retrospective prognostic evaluation using unsupervised learning in the treatment of COVID-19 patients with hypertension treated with ACEI/ARB drugs

Introduction

This study aimed to evaluate the prognosis of patients with COVID-19 and hypertension who were treated with angiotensin-converting enzyme inhibitor (ACEI)/angiotensin receptor B (ARB) drugs and to identify key features affecting patient prognosis using an unsupervised learning method.

Methods

A large-scale clinical dataset, including patient information, medical history, and laboratory test results, was collected. Two hundred patients with COVID-19 and hypertension were included. After cluster analysis, patients were divided into good and poor prognosis groups. The unsupervised learning method was used to evaluate clinical characteristics and prognosis, and patients were divided into different prognosis groups. The improved wild dog optimization algorithm (IDOA) was used for feature selection and cluster analysis, followed by the IDOA-k-means algorithm. The impact of ACEI/ARB drugs on patient prognosis and key characteristics affecting patient prognosis were also analysed.

Results

Key features related to prognosis included baseline information and laboratory test results, while clinical symptoms and imaging results had low predictive power. The top six important features were age, hypertension grade, MuLBSTA, ACEI/ARB, NT-proBNP, and high-sensitivity troponin I. These features were consistent with the results of the unsupervised prediction model. A visualization system was developed based on these key features.

Conclusion

Using unsupervised learning and the improved k-means algorithm, this study accurately analysed the prognosis of patients with COVID-19 and hypertension. The use of ACEI/ARB drugs was found to be a protective factor for poor clinical prognosis. Unsupervised learning methods can be used to differentiate patient populations and assess treatment effects. This study identified important features affecting patient prognosis and developed a visualization system with clinical significance for prognosis assessment and treatment decision-making.

A Comprehensive Survey on Deep Graph Representation Learning

Graph representation learning aims to effectively encode high-dimensional sparse graph-structured data into low-dimensional dense vectors, which is a fundamental task that has been widely studied in a range of fields, including machine learning and data mining. Classic graph embedding methods follow the basic idea that the embedding vectors of interconnected nodes in the graph can still maintain a relatively close distance, thereby preserving the structural information between the nodes in the graph. However, this is sub-optimal due to: (i) traditional methods have limited model capacity which limits the learning performance; (ii) existing techniques typically rely on unsupervised learning strategies and fail to couple with the latest learning paradigms; (iii) representation learning and downstream tasks are dependent on each other which should be jointly enhanced. With the remarkable success of deep learning, deep graph representation learning has shown great potential and advantages over shallow (traditional) methods, there exist a large number of deep graph representation learning techniques have been proposed in the past decade, especially graph neural networks. In this survey, we conduct a comprehensive survey on current deep graph representation learning algorithms by proposing a new taxonomy of existing state-of-the-art literature. Specifically, we systematically summarize the essential components of graph representation learning and categorize existing approaches by the ways of graph neural network architectures and the most recent advanced learning paradigms. Moreover, this survey also provides the practical and promising applications of deep graph representation learning. Last but not least, we state new perspectives and suggest challenging directions which deserve further investigations in the future.

Towards generalizable Graph Contrastive Learning: An information theory perspective

Graph Contrastive Learning (GCL) is increasingly employed in graph representation learning with the primary aim of learning node/graph representations from a predefined pretext task that can generalize to various downstream tasks. Meanwhile, the transition from a specific pretext task to diverse and unpredictable downstream tasks poses a significant challenge for GCL's generalization ability. Most existing GCL approaches maximize mutual information between two views derived from the original graph, either randomly or heuristically. However, the generalization ability of GCL and its theoretical principles are still less studied. In this paper, we introduce a novel metric GCL-GE, to quantify the generalization gap between predefined pretext and agnostic downstream tasks. Given the inherent intractability of GCL-GE, we leverage concepts from information theory to derive a mutual information upper bound that is independent of the downstream tasks, thus enabling the metric's optimization despite the variability in downstream tasks. Based on the theoretical insight, we propose InfoAdv, a GCL framework to directly enhance generalization by jointly optimizing GCL-GE and InfoMax. Extensive experiments validate the capability of InfoAdv to enhance performance across a wide variety of downstream tasks, demonstrating its effectiveness in improving the generalizability of GCL.

Object-stable unsupervised dual contrastive learning image-to-image translation with query-selected attention and convolutional block attention module

Recently, contrastive learning has gained popularity in the field of unsupervised image-to-image (I2I) translation. In a previous study, a query-selected attention (QS-Attn) module, which employed an attention matrix with a probability distribution, was used to maximize the mutual information between the source and translated images. This module selected significant queries using an entropy metric computed from the attention matrix. However, it often selected many queries with equal significance measures, leading to an excessive focus on the background. In this study, we proposed a dual-learning framework with QS-Attn and convolutional block attention module (CBAM) called object-stable dual contrastive learning generative adversarial network (OS-DCLGAN). In this paper, we utilize a CBAM, which learns what and where to emphasize or suppress, thereby refining intermediate features effectively. This CBAM was integrated before the QS-Attn module to capture significant domain information for I2I translation tasks. The proposed framework outperformed recently introduced approaches in various I2I translation tasks, showing its effectiveness and versatility. The code is available at https://github.com/RedPotatoChip/OSUDL.

Self-supervised contrastive graph representation with node and graph augmentation

Graph representation is a critical technology in the field of knowledge engineering and knowledge-based applications since most knowledge bases are represented in the graph structure. Nowadays, contrastive learning has become a prominent way for graph representation by contrasting positive-positive and positive-negative node pairs between two augmentation graphs. It has achieved new state-of-the-art in the field of self-supervised graph representation. However, existing contrastive graph representation methods mainly focus on modifying (normally removing some edges/nodes) the original graph structure to generate the augmentation graph for the contrastive. It inevitably changes the original graph structures, meaning the generated augmentation graph is no longer equivalent to the original graph. This harms the performance of the representation in many structure-sensitive graphs such as protein graphs, chemical graphs, molecular graphs, etc. Moreover, there is only one positive-positive node pair but relatively massive positive-negative node pairs in the self-supervised graph contrastive learning. This can lead to the same class, or very similar samples are considered negative samples. To this end, in this work, we propose a Virtual Masking Augmentation (VMA) to generate an augmentation graph without changing any structures from the original graph. Meanwhile, a node augmentation method is proposed to augment the positive node pairs by discovering the most similar nodes in the same graph. Then, two different augmentation graphs are generated and put into a contrastive learning model to learn the graph representation. Extensive experiments on massive datasets demonstrate that our method achieves new state-of-the-art results on self-supervised graph representation. The source code of the proposed method is available at https://github.com/DuanhaoranCC/CGRA.

3D graph neural network with few-shot learning for predicting drug-drug interactions in scaffold-based cold start scenario

Understanding drug-drug interactions (DDI) of new drugs is critical for minimizing unexpected adverse drug reactions. The modeling of new drugs is called a cold start scenario. In this scenario, Only a few structural information or physicochemical information about new drug is available. The 3D conformation of drug molecules usually plays a crucial role in chemical properties compared to the 2D structure. 3D graph network with few-shot learning is a promising solution. However, the 3D heterogeneity of drug molecules and the discretization of atomic distributions lead to spatial confusion in few-shot learning. Here, we propose a 3D graph neural network with few-shot learning, Meta3D-DDI, to predict DDI events in cold start scenario. The 3DGNN ensures rotation and translation invariance by calculating atomic pairwise distances, and incorporates 3D structure and distance information in the information aggregation stage. The continuous filter interaction module can continuously simulate the filter to obtain the interaction between the target atom and other atoms. Meta3D-DDI further develops a FSL strategy based on bilevel optimization to transfer meta-knowledge for DDI prediction tasks from existing drugs to new drugs. In addition, the existing cold start setting may cause the scaffold structure information in the training set to leak into the test set. We design scaffold-based cold start scenario to ensure that the drug scaffolds in the training set and test set do not overlap. The extensive experiments demonstrate that our architecture achieves the SOTA performance for DDI prediction under scaffold-based cold start scenario on two real-world datasets. The visual experiment shows that Meta3D-DDI significantly improves the learning for DDI prediction of new drugs. We also demonstrate how Meta3D-DDI can reduce the amount of data required to make meaningful DDI predictions.