Dynamic Embedding Projection-Gated Convolutional Neural Networks for Text Classification

Achieving Sharp Upper Bounds on the Expressive Power of Neural Networks via Tropical Polynomials

The expressive power of neural networks describes the ability to represent or approximate complex functions. The number of linear regions is the standard and most natural measure of expressive power. However, a major challenge in utilizing the number of linear regions as a measure of expressive power is the exponential gap between the theoretical upper and lower bounds, which becomes more pronounced as the neural network capacity increases. In this article, we aim to derive a sharp upper bound on piecewise linear neural networks (PLNNs) to bridge this gap. Specifically, we first establish the relationship between tropical polynomials and PLNNs. In the unexpanded tropical polynomials form, we make the proposition that hyperplanes are not all in the general positions, thereby reducing the number of intersecting hyperplanes. We propose a rank-based approach and present the empirical analysis that this approach outperforms previous Zaslavsky's theorem-based methods. In the expanded tropical polynomials form, accounting for limitations in weight initialization and model computational precision, we raise the concept that the values range of each term is bounded. We propose a precision-based approach that transforms the approximate exponential growth of the number of linear regions into polynomial growth with width, which is effective at larger layer widths. Finally, we compare the number of linear regions that can be represented by each hidden layer in both forms and derive a sharp upper bound for PLNNs. Empirical analysis and experimental results provide compelling evidence for the efficacy and feasibility of this sharp upper bound on both simulated experiments and real datasets.

CoGraphNet for enhanced text classification using word-sentence heterogeneous graph representations and improved interpretability

Text Graph Representation Learning through Graph Neural Networks (TG-GNN) is a powerful approach in natural language processing and information retrieval. However, it faces challenges in computational complexity and interpretability. In this work, we propose CoGraphNet, a novel graph-based model for text classification, addressing key issues. To overcome information loss, we construct separate heterogeneous graphs for words and sentences, capturing multi-tiered contextual information. We enhance interpretability by incorporating positional bias weights, improving model clarity. CoGraphNet provides precise analysis, highlighting important words or sentences. We achieve enhanced contextual comprehension and accuracy through novel graph structures and the SwiGLU activation function. Experiments on Ohsumed, MR, R52, and 20NG datasets confirm CoGraphNet's effectiveness in complex classification tasks, demonstrating its superiority.

A framework for hardware trojan detection based on contrastive learning

With the rapid development of the semiconductor industry, Hardware Trojans (HT) as a kind of malicious function that can be implanted at will in all processes of integrated circuit design, manufacturing, and deployment have become a great threat in the field of hardware security. Side-channel analysis is widely used in the detection of HT due to its high efficiency, non-contact nature, and accuracy. In this paper, we propose a framework for HT detection based on contrastive learning using power consumption information in unsupervised or weakly supervised scenarios. First, the framework augments the data, such as creatively using a one-dimensional discrete chaotic mapping to disturb the data to achieve data augmentation to improve the generalization capabilities of the model. Second, the model representation is learned by comparing the similarities and differences between samples, freeing it from the dependence on labels. Finally, the detection of HT is accomplished more efficiently by categorizing the side information during circuit operation through the backbone network. Experiments on data from nine different public HTs show that the proposed method exhibits better generalization capabilities using the same network model within a comparative learning framework. The model trained on the dataset of small Trojan T100 has a detection efficiency advantage of up to 44% in detecting large Trojans, while the model trained on the dataset of large Trojan T2100 has a detection efficiency advantage of up to 10% in detecting small Trojans. The results in data imbalanced and noisy environments also show that the contrastive learning framework in this paper can better fulfill the requirements of detecting unknown HT in unsupervised or weakly supervised scenarios.

Decision model of public opinion risk in campus social network based on hybrid dynamic deletion and shortest path algorithm

Aiming at the problem that traditional network public opinion monitoring and searching are inefficient and can easily cause resource waste, the study firstly, through the dynamic deletion-shortest path algorithm to classify network text, and on this basis, innovatively constructs a text sentiment classification model based on the variant of convolutional neural network and recurrent neural network, and secondly, uses attention mechanism to classify the model. improvement of the classification model by using the attention mechanism. The research results show that the average precision rate, recall rate, and F-value of the dynamic deletion-shortest path algorithm are 97.30%, 79.55%, and 87.53%, and the classification speed is 397 KB/s, which is better than the traditional shortest path algorithm. In the classification effect measurement of long text, the accuracy and F-value of the recurrent neural network variant model are above 84%, and the accuracy of the text sentiment classification model with the introduction of the attention mechanism is improved by 3.89% compared to the pre-improvement period. In summary, the dynamic deletion-shortest path algorithm proposed in the study and the sentiment classification model with the introduction of the attention mechanism have superior performance and can provide certain application value for campus social network opinion risk decision-making.

Emotion-Semantic-Aware Dual Contrastive Learning for Epistemic Emotion Identification of Learner-Generated Reviews in MOOCs

Identifying the epistemic emotions of learner-generated reviews in massive open online courses (MOOCs) can help instructors provide adaptive guidance and interventions for learners. The epistemic emotion identification task is a fine-grained identification task that contains multiple categories of emotions arising during the learning process. Previous studies only consider emotional or semantic information within the review texts alone, which leads to insufficient feature representation. In addition, some categories of epistemic emotions are ambiguously distributed in feature space, making them hard to be distinguished. In this article, we present an emotion-semantic-aware dual contrastive learning (ES-DCL) approach to tackle these issues. In order to learn sufficient feature representation, implicit semantic features and human-interpretable emotional features are, respectively, extracted from two different views to form complementary emotional-semantic features. On this basis, by leveraging the experience of domain experts and the input emotional-semantic features, two types of contrastive losses (label contrastive loss and feature contrastive loss) are formulated. They are designed to train the discriminative distribution of emotional-semantic features in the sample space and to solve the anisotropy problem between different categories of epistemic emotions. The proposed ES-DCL is compared with 11 other baseline models on four different disciplinary MOOCs review datasets. Extensive experimental results show that our approach improves the performance of epistemic emotion identification, and significantly outperforms state-of-the-art deep learning-based methods in learning more discriminative sentence representations.

Description-Enhanced Label Embedding Contrastive Learning for Text Classification

Text classification is one of the fundamental tasks in natural language processing, which requires an agent to determine the most appropriate category for input sentences. Recently, deep neural networks have achieved impressive performance in this area, especially pretrained language models (PLMs). Usually, these methods concentrate on input sentences and corresponding semantic embedding generation. However, for another essential component: labels, most existing works either treat them as meaningless one-hot vectors or use vanilla embedding methods to learn label representations along with model training, underestimating the semantic information and guidance that these labels reveal. To alleviate this problem and better exploit label information, in this article, we employ self-supervised learning (SSL) in model learning process and design a novel self-supervised relation of relation ( [Formula: see text]) classification task for label utilization from a one-hot manner perspective. Then, we propose a novel relation of relation learning network( [Formula: see text]-Net) for text classification, in which text classification and [Formula: see text] classification are treated as optimization targets. Meanwhile, triplet loss is employed to enhance the analysis of differences and connections among labels. Moreover, considering that one-hot usage is still short of exploiting label information, we incorporate external knowledge from WordNet to obtain multiaspect descriptions for label semantic learning and extend [Formula: see text]-Net to a novel description-enhanced label embedding network(DELE) from a label embedding perspective. One step further, since these fine-grained descriptions may introduce unexpected noise, we develop a mutual interaction module to select appropriate parts from input sentences and labels simultaneously based on contrastive learning (CL) for noise mitigation. Extensive experiments on different text classification tasks reveal that [Formula: see text]-Net can effectively improve the classification performance and DELE can make better use of label information and further improve the performance. As a byproduct, we have released the codes to facilitate other research.

Living review framework for better policy design and management of hazardous waste in Australia

The significant increase in hazardous waste generation in Australia has led to the discussion over the incorporation of artificial intelligence into the hazardous waste management system. Recent studies explored the potential applications of artificial intelligence in various processes of managing waste. However, no study has examined the use of text mining in the hazardous waste management sector for the purpose of informing policymakers. This study developed a living review framework which applied supervised text classification and text mining techniques to extract knowledge using the domain literature data between 2022 and 2023. The framework employed statistical classification models trained using iterative training and the best model XGBoost achieved an F1 score of 0.87. Using a small set of 126 manually labelled global articles, XGBoost automatically predicted the labels of 678 Australian articles with high confidence. Then, keyword extraction and unsupervised topic modelling with Latent Dirichlet Allocation (LDA) were performed. Results indicated that there were 2 main research themes in Australian literature: (1) the key waste streams and (2) the resource recovery and recycling of waste. The implication of this framework would benefit the policymakers, researchers, and hazardous waste management organisations by serving as a real time guideline of the current key waste streams and research themes in the literature which allow robust knowledge to be applied to waste management and highlight where the gap in research remains.

Research on performance variations of classifiers with the influence of pre-processing methods for Chinese short text classification

Text pre-processing is an important component of a Chinese text classification. At present, however, most of the studies on this topic focus on exploring the influence of preprocessing methods on a few text classification algorithms using English text. In this paper we experimentally compared fifteen commonly used classifiers on two Chinese datasets using three widely used Chinese preprocessing methods that include word segmentation, Chinese specific stop word removal, and Chinese specific symbol removal. We then explored the influence of the preprocessing methods on the final classifications according to various conditions such as classification evaluation, combination style, and classifier selection. Finally, we conducted a battery of various additional experiments, and found that most of the classifiers improved in performance after proper preprocessing was applied. Our general conclusion is that the systematic use of preprocessing methods can have a positive impact on the classification of Chinese short text, using classification evaluation such as macro-F1, combination of preprocessing methods such as word segmentation, Chinese specific stop word and symbol removal, and classifier selection such as machine and deep learning models. We find that the best macro-f1s for categorizing text for the two datasets are 92.13% and 91.99%, which represent improvements of 0.3% and 2%, respectively over the compared baselines.

Variational gated autoencoder-based feature extraction model for inferring disease-miRNA associations based on multiview features

MicroRNAs (miRNA) play critical roles in diverse biological processes of diseases. Inferring potential disease-miRNA associations enable us to better understand the development and diagnosis of complex human diseases via computational algorithms. The work presents a variational gated autoencoder-based feature extraction model to extract complex contextual features for inferring potential disease-miRNA associations. Specifically, our model fuses three different similarities of miRNAs into a comprehensive miRNA network and then combines two various similarities of diseases into a comprehensive disease network, respectively. Then, a novel graph autoencoder is designed to extract multilevel representations based on variational gate mechanisms from heterogeneous networks of miRNAs and diseases. Finally, a gate-based association predictor is devised to combine multiscale representations of miRNAs and diseases via a novel contrastive cross-entropy function, and then infer disease-miRNA associations. Experimental results indicate that our proposed model achieves remarkable association prediction performance, proving the efficacy of the variational gate mechanism and contrastive cross-entropy loss for inferring disease-miRNA associations.

Federal learning edge network based sentiment analysis combating global COVID-19

As one of the important research topics in the field of natural language processing, sentiment analysis aims to analyze web data related to COVID-19, e.g., supporting China government agencies combating COVID-19. There are popular sentiment analysis models based on deep learning techniques, but their performance is limited by the size and distribution of the dataset. In this study, we propose a model based on a federal learning framework with Bert and multi-scale convolutional neural network (Fed_BERT_MSCNN), which contains a Bidirectional Encoder Representations from Transformer modules and a multi-scale convolution layer. The federal learning framework contains a central server and local deep learning machines that train local datasets. Parameter communications were processed through edge networks. The weighted average of each participant's model parameters was communicated in the edge network for final utilization. The proposed federal network not only solves the problem of insufficient data, but also ensures the data privacy of the social platform during the training process and improve the communication efficiency. In the experiment, we used datasets of six social platforms, and used accuracy and F1-score as evaluation criteria to conduct comparative studies. The performance of the proposed Fed_BERT_MSCNN model was generally superior than the existing models in the literature.

Graph-in-Graph Convolutional Network for Hyperspectral Image Classification

With the development of hyperspectral sensors, accessible hyperspectral images (HSIs) are increasing, and pixel-oriented classification has attracted much attention. Recently, graph convolutional networks (GCNs) have been proposed to process graph-structured data in non-Euclidean domains and have been employed in HSI classification. But most methods based on GCN are hard to sufficiently exploit information of ground objects due to feature aggregation. To solve this issue, in this article, we proposed a graph-in-graph (GiG) model and a related GiG convolutional network (GiGCN) for HSI classification from a superpixel viewpoint. The GiG representation covers information inside and outside superpixels, respectively, corresponding to the local and global characteristics of ground objects. Concretely, after segmenting HSI into disjoint superpixels, each one is converted to an internal graph. Meanwhile, an external graph is constructed according to the spatial adjacent relationships among superpixels. Significantly, each node in the external graph embeds a corresponding internal graph, forming the so-called GiG structure. Then, GiGCN composed of internal and External graph convolution (EGC) is designed to extract hierarchical features and integrate them into multiple scales, improving the discriminability of GiGCN. Ensemble learning is incorporated to further boost the robustness of GiGCN. It is worth noting that we are the first to propose the GiG framework from the superpixel point and the GiGCN scheme for HSI classification. Experiment results on four benchmark datasets demonstrate that our proposed method is effective and feasible for HSI classification with limited labeled samples. For study replication, the code developed for this study is available at https://github.com/ShuGuoJ/GiGCN.git.

A Deep Fusion Matching Network Semantic Reasoning Model

As the vital technology of natural language understanding, sentence representation reasoning technology mainly focuses on sentence representation methods and reasoning models. Although the performance has been improved, there are still some problems, such as incomplete sentence semantic expression, lack of depth of reasoning model, and lack of interpretability of the reasoning process. Given the reasoning model’s lack of reasoning depth and interpretability, a deep fusion matching network is designed in this paper, which mainly includes a coding layer, matching layer, dependency convolution layer, information aggregation layer, and inference prediction layer. Based on a deep matching network, the matching layer is improved. Furthermore, the heuristic matching algorithm replaces the bidirectional long-short memory neural network to simplify the interactive fusion. As a result, it improves the reasoning depth and reduces the complexity of the model; the dependency convolution layer uses the tree-type convolution network to extract the sentence structure information along with the sentence dependency tree structure, which improves the interpretability of the reasoning process. Finally, the performance of the model is verified on several datasets. The results show that the reasoning effect of the model is better than that of the shallow reasoning model, and the accuracy rate on the SNLI test set reaches 89.0%. At the same time, the semantic correlation analysis results show that the dependency convolution layer is beneficial in improving the interpretability of the reasoning process.

A Study of Text Vectorization Method Combining Topic Model and Transfer Learning

With the development of Internet cloud technology, the scale of data is expanding. Traditional processing methods find it difficult to deal with the problem of information extraction of big data. Therefore, it is necessary to use machine-learning-assisted intelligent processing to extract information from data in order to solve the optimization problem in complex systems. There are many forms of data storage. Among them, text data is an important data type that directly reflects semantic information. Text vectorization is an important concept in natural language processing tasks. Because text data can not be directly used for model parameter training, it is necessary to vectorize the original text data and make it numerical, and then the feature extraction operation can be carried out. The traditional text digitization method is often realized by constructing a bag of words, but the vector generated by this method can not reflect the semantic relationship between words, and it also easily causes the problems of data sparsity and dimension explosion. Therefore, this paper proposes a text vectorization method combining a topic model and transfer learning. Firstly, the topic model is selected to model the text data and extract its keywords, to grasp the main information of the text data. Then, with the help of the bidirectional encoder representations from transformers (BERT) model, which belongs to the pretrained model, model transfer learning is carried out to generate vectors, which are applied to the calculation of similarity between texts. By setting up a comparative experiment, this method is compared with the traditional vectorization method. The experimental results show that the vector generated by the topic-modeling- and transfer-learning-based text vectorization (TTTV) proposed in this paper can obtain better results when calculating the similarity between texts with the same topic, which means that it can more accurately judge whether the contents of the given two texts belong to the same topic.

Bert-Enhanced Text Graph Neural Network for Classification

Text classification is a fundamental research direction, aims to assign tags to text units. Recently, graph neural networks (GNN) have exhibited some excellent properties in textual information processing. Furthermore, the pre-trained language model also realized promising effects in many tasks. However, many text processing methods cannot model a single text unit’s structure or ignore the semantic features. To solve these problems and comprehensively utilize the text’s structure information and semantic information, we propose a Bert-Enhanced text Graph Neural Network model (BEGNN). For each text, we construct a text graph separately according to the co-occurrence relationship of words and use GNN to extract text features. Moreover, we employ Bert to extract semantic features. The former part can take into account the structural information, and the latter can focus on modeling the semantic information. Finally, we interact and aggregate these two features of different granularity to get a more effective representation. Experiments on standard datasets demonstrate the effectiveness of BEGNN.

Sentiment classification for employees reviews using regression vector- stochastic gradient descent classifier (RV-SGDC)

The satisfaction of employees is very important for any organization to make sufficient progress in production and to achieve its goals. Organizations try to keep their employees satisfied by making their policies according to employees' demands which help to create a good environment for the collective. For this reason, it is beneficial for organizations to perform staff satisfaction surveys to be analyzed, allowing them to gauge the levels of satisfaction among employees. Sentiment analysis is an approach that can assist in this regard as it categorizes sentiments of reviews into positive and negative results. In this study, we perform experiments for the world's big six companies and classify their employees' reviews based on their sentiments. For this, we proposed an approach using lexicon-based and machine learning based techniques. Firstly, we extracted the sentiments of employees from text reviews and labeled the dataset as positive and negative using TextBlob. Then we proposed a hybrid/voting model named Regression Vector-Stochastic Gradient Descent Classifier (RV-SGDC) for sentiment classification. RV-SGDC is a combination of logistic regression, support vector machines, and stochastic gradient descent. We combined these models under a majority voting criteria. We also used other machine learning models in the performance comparison of RV-SGDC. Further, three feature extraction techniques: term frequency-inverse document frequency (TF-IDF), bag of words, and global vectors are used to train learning models. We evaluated the performance of all models in terms of accuracy, precision, recall, and F1 score. The results revealed that RV-SGDC outperforms with a 0.97 accuracy score using the TF-IDF feature due to its hybrid architecture.