Transfer Learning for Sentiment Analysis Using BERT Based Supervised Fine-Tuning

Domain adaptive learning for multi realm sentiment classification on big data

Machine learning techniques that rely on textual features or sentiment lexicons can lead to erroneous sentiment analysis. These techniques are especially vulnerable to domain-related difficulties, especially when dealing in Big data. In addition, labeling is time-consuming and supervised machine learning algorithms often lack labeled data. Transfer learning can help save time and obtain high performance with fewer datasets in this field. To cope this, we used a transfer learning-based Multi-Domain Sentiment Classification (MDSC) technique. We are able to identify the sentiment polarity of text in a target domain that is unlabeled by looking at reviews in a labelled source domain. This research aims to evaluate the impact of domain adaptation and measure the extent to which transfer learning enhances sentiment analysis outcomes. We employed transfer learning models BERT, RoBERTa, ELECTRA, and ULMFiT to improve the performance in sentiment analysis. We analyzed sentiment through various transformer models and compared the performance of LSTM and CNN. The experiments are carried on five publicly available sentiment analysis datasets, namely Hotel Reviews (HR), Movie Reviews (MR), Sentiment140 Tweets (ST), Citation Sentiment Corpus (CSC), and Bioinformatics Citation Corpus (BCC), to adapt multi-target domains. The performance of numerous models employing transfer learning from diverse datasets demonstrating how various factors influence the outputs.

The characteristics of online gerontophobia expressions in South Korea

Recently, South Korea has been transitioning into a super-aged society. The purpose of this paper is to identify the patterns and underlying causes of gerontophobia expressions in South Korea. This paper refines the patterns of gerontophobia expressions into five types: "Fear of Aging," "Resource Burden," "Social Isolation," "Criticism of Social Behavior," and "Stereotypes of Political Orientation." Based on these types, this study develops a deep learning algorithm to detect the type of gerontophobia expressions. To do this, kc-BERT was used and 760,140 news comments (for six years from May 1, 2017, to June 31, 2021) in Naver news was used. The result shows that "Fear of Aging" type exhibited a significant decreasing trend, while the other types showed no meaningful changes. The results of topic modeling on news articles indicated that various aspects of elderly life, unresolved historical events, COVID-19, digital and financial exclusion, economic and social welfare, and other critical societal issues co-occur and contribute to gerontophobia. This study provides a framework to understand the characteristics of online gerontophobia, offering insights into its underlying causes, and providing practical implications for policy makers.

OpenDeID Pipeline for Unstructured Electronic Health Record Text Notes Based on Rules and Transformers: Deidentification Algorithm Development and Validation Study

BACKGROUND

Electronic health records (EHRs) in unstructured formats are valuable sources of information for research in both the clinical and biomedical domains. However, before such records can be used for research purposes, sensitive health information (SHI) must be removed in several cases to protect patient privacy. Rule-based and machine learning-based methods have been shown to be effective in deidentification. However, very few studies investigated the combination of transformer-based language models and rules.

OBJECTIVE

The objective of this study is to develop a hybrid deidentification pipeline for Australian EHR text notes using rules and transformers. The study also aims to investigate the impact of pretrained word embedding and transformer-based language models.

METHODS

In this study, we present a hybrid deidentification pipeline called OpenDeID, which is developed using an Australian multicenter EHR-based corpus called OpenDeID Corpus. The OpenDeID corpus consists of 2100 pathology reports with 38,414 SHI entities from 1833 patients. The OpenDeID pipeline incorporates a hybrid approach of associative rules, supervised deep learning, and pretrained language models.

RESULTS

The OpenDeID achieved a best F1-score of 0.9659 by fine-tuning the Discharge Summary BioBERT model and incorporating various preprocessing and postprocessing rules. The OpenDeID pipeline has been deployed at a large tertiary teaching hospital and has processed over 8000 unstructured EHR text notes in real time.

CONCLUSIONS

The OpenDeID pipeline is a hybrid deidentification pipeline to deidentify SHI entities in unstructured EHR text notes. The pipeline has been evaluated on a large multicenter corpus. External validation will be undertaken as part of our future work to evaluate the effectiveness of the OpenDeID pipeline.

Transfer Learning for Sentiment Classification Using Bidirectional Encoder Representations from Transformers (BERT) Model

Sentiment is currently one of the most emerging areas of research due to the large amount of web content coming from social networking websites. Sentiment analysis is a crucial process for recommending systems for most people. Generally, the purpose of sentiment analysis is to determine an author's attitude toward a subject or the overall tone of a document. There is a huge collection of studies that make an effort to predict how useful online reviews will be and have produced conflicting results on the efficacy of different methodologies. Furthermore, many of the current solutions employ manual feature generation and conventional shallow learning methods, which restrict generalization. As a result, the goal of this research is to develop a general approach using transfer learning by applying the "BERT (Bidirectional Encoder Representations from Transformers)"-based model. The efficiency of BERT classification is then evaluated by comparing it with similar machine learning techniques. In the experimental evaluation, the proposed model demonstrated superior performance in terms of outstanding prediction and high accuracy compared to earlier research. Comparative tests conducted on positive and negative Yelp reviews reveal that fine-tuned BERT classification performs better than other approaches. In addition, it is observed that BERT classifiers using batch size and sequence length significantly affect classification performance.

Sentiment analysis of Indonesian datasets based on a hybrid deep-learning strategy

Various attempts have been conducted to improve the performance of text-based sentiment analysis. These significant attempts have focused on text representation and model classifiers. This paper introduced a hybrid model based on the text representation and the classifier models, to address sentiment classification with various topics. The combination of BERT and a distilled version of BERT (DistilBERT) was selected in the representative vectors of the input sentences, while the combination of long short-term memory and temporal convolutional networks was taken to enhance the proposed model in understanding the semantics and context of each word. The experiment results showed that the proposed model outperformed various counterpart schemes in considered metrics. The reliability of the proposed model was confirmed in a mixed dataset containing nine topics.

Textual emotion detection utilizing a transfer learning approach

Many attempts have been made to overcome the challenges of automating textual emotion detection using different traditional deep learning models such as LSTM, GRU, and BiLSTM. But the problem with these models is that they need large datasets, massive computing resources, and a lot of time to train. Also, they are prone to forgetting and cannot perform well when applied to small datasets. In this paper, we aim to demonstrate the capability of transfer learning techniques to capture the better contextual meaning of the text and as a result better detection of the emotion represented in the text, even without a large amount of data and training time. To do this, we conduct an experiment utilizing a pre-trained model called EmotionalBERT, which is based on bidirectional encoder representations from transformers (BERT), and we compare its performance to RNN-based models on two benchmark datasets, with a focus on the amount of training data and how it affects the models' performance.

Learned Text Representation for Amharic Information Retrieval and Natural Language Processing

Over the past few years, word embeddings and bidirectional encoder representations from transformers (BERT) models have brought better solutions to learning text representations for natural language processing (NLP) and other tasks. Many NLP applications rely on pre-trained text representations, leading to the development of a number of neural network language models for various languages. However, this is not the case for Amharic, which is known to be a morphologically complex and under-resourced language. Usable pre-trained models for automatic Amharic text processing are not available. This paper presents an investigation on the essence of learned text representation for information retrieval and NLP tasks using word embeddings and BERT language models. We explored the most commonly used methods for word embeddings, including word2vec, GloVe, and fastText, as well as the BERT model. We investigated the performance of query expansion using word embeddings. We also analyzed the use of a pre-trained Amharic BERT model for masked language modeling, next sentence prediction, and text classification tasks. Amharic ad hoc information retrieval test collections that contain word-based, stem-based, and root-based text representations were used for evaluation. We conducted a detailed empirical analysis on the usability of word embeddings and BERT models on word-based, stem-based, and root-based corpora. Experimental results show that word-based query expansion and language modeling perform better than stem-based and root-based text representations, and fastText outperforms other word embeddings on word-based corpus.

Understanding Quality of Products from Customers’ Attitude Using Advanced Machine Learning Methods

The trend of E-commerce and online shopping is increasing rapidly. However, it is difficult to know about the quality of items from pictures and videos available on the online stores. Therefore, online stores and independent products reviews sites share user reviews about the products for the ease of buyers to find out the best quality products. The proposed work is about measuring and detecting product quality based on consumers’ attitude in product reviews. Predicting the quality of a product from customers’ reviews is a challenging and novel research area. Natural Language Processing and machine learning methods are popularly employed to identify product quality from customer reviews. Most of the existing research for the product review system has been done using traditional sentiment analysis and opinion mining. Going beyond the constraints of opinion and sentiment, such as a deeper description of the input text, is made possible by utilizing appraisal categories. The main focus of this study is exploiting the quality subcategory of the appraisal framework in order to predict the quality of the product. This paper presents a quality of product-based classification model (named QLeBERT) by combining quality of product-related lexicon, N-grams, Bidirectional Encoder Representations from Transformers (BERT), and Bidirectional Long Short Term Memory (BiLSTM). In the proposed model, the quality of the product-related lexicon, N-grams, and BERT are employed to generate vectors of words from part of the customers’ reviews. The main contribution of this work is the preparation of the quality of product-related lexicon dictionary based on an appraisal framework and automatically labelling the data accordingly before using them as the training data in the BiLSTM model. The proposed model is evaluated on an Amazon product reviews dataset. The proposed QLeBERT outperforms the existing state-of-the-art models by achieving an F1macro score of 0.91 in binary classification.

Fake News Detection Model on Social Media by Leveraging Sentiment Analysis of News Content and Emotion Analysis of Users' Comments

Nowadays, social media has become the main source of news around the world. The spread of fake news on social networks has become a serious global issue, damaging many aspects, such as political, economic, and social aspects, and negatively affecting the lives of citizens. Fake news often carries negative sentiments, and the public's response to it carries the emotions of surprise, fear, and disgust. In this article, we extracted features based on sentiment analysis of news articles and emotion analysis of users' comments regarding this news. These features were fed, along with the content feature of the news, to the proposed bidirectional long short-term memory model to detect fake news. We used the standard Fakeddit dataset that contains news titles and comments posted regarding them to train and test the proposed model. The suggested model, using extracted features, provided a high detection accuracy of 96.77% of the Area under the ROC Curve measure, which is higher than what other state-of-the-art studies offer. The results prove that the features extracted based on sentiment analysis of news, which represents the publisher's stance, and emotion analysis of comments, which represent the crowd's stance, contribute to raising the efficiency of the detection model.

What people think about fast food: opinions analysis and LDA modeling on fast food restaurants using unstructured tweets

With the rise of social media platforms, sharing reviews has become a social norm in today's modern society. People check customer views on social networking sites about different fast food restaurants and food items before visiting the restaurants and ordering food. Restaurants can compete to better the quality of their offered items or services by carefully analyzing the feedback provided by customers. People tend to visit restaurants with a higher number of positive reviews. Accordingly, manually collecting feedback from customers for every product is a labor-intensive process; the same is true for sentiment analysis. To overcome this, we use sentiment analysis, which automatically extracts meaningful information from the data. Existing studies predominantly focus on machine learning models. As a consequence, the performance analysis of deep learning models is neglected primarily and of the deep ensemble models especially. To this end, this study adopts several deep ensemble models including Bi long short-term memory and gated recurrent unit (BiLSTM+GRU), LSTM+GRU, GRU+recurrent neural network (GRU+RNN), and BiLSTM+RNN models using self-collected unstructured tweets. The performance of lexicon-based methods is compared with deep ensemble models for sentiment classification. In addition, the study makes use of Latent Dirichlet Allocation (LDA) modeling for topic analysis. For experiments, the tweets for the top five fast food serving companies are collected which include KFC, Pizza Hut, McDonald's, Burger King, and Subway. Experimental results reveal that deep ensemble models yield better results than the lexicon-based approach and BiLSTM+GRU obtains the highest accuracy of 95.31% for three class problems. Topic modeling indicates that the highest number of negative sentiments are represented for Subway restaurants with high-intensity negative words. The majority of the people (49%) remain neutral regarding the choice of fast food, 31% seem to like fast food while the rest (20%) dislike fast food.

End-to-End Transformer-Based Models in Textual-Based NLP

Transformer architectures are highly expressive because they use self-attention mechanisms to encode long-range dependencies in the input sequences. In this paper, we present a literature review on Transformer-based (TB) models, providing a detailed overview of each model in comparison to the Transformer’s standard architecture. This survey focuses on TB models used in the field of Natural Language Processing (NLP) for textual-based tasks. We begin with an overview of the fundamental concepts at the heart of the success of these models. Then, we classify them based on their architecture and training mode. We compare the advantages and disadvantages of popular techniques in terms of architectural design and experimental value. Finally, we discuss open research, directions, and potential future work to help solve current TB application challenges in NLP.