Image captioning by incorporating affective concepts learned from both visual and textual components

Design of Service Robot Based on User Emotion Recognition and Environmental Monitoring

Robots may be able to comprehend human emotions better by adding speech emotion recognition and environment monitoring functions to human-computer interaction systems. Robots can offer more humanized services by adapting to human emotions, resulting in a comfortable and cordial interaction between humans and robots. Improve the environment for communication and computer-human interaction and also interactive computer experience. In order for service robots to perform fluid human-computer interaction, this paper designs a sentiment analysis model based on CNN (convective neural network) to detect the feeling of interacting objects. It also builds a sentiment analysis model and an open domain dialogue system suitable for service robots. Examine the emotions experienced by the objects while they conversed. According to test results, the sentiment classification method used in this article performs more accurately on the dataset than the conventional model, and the final sentiment analysis model's F1 value can reach 0.931, which is better able to identify an emotional state. Using all voice samples as the input content of the network would eliminate the confusion between neutral emotions and other nonneutral emotions, boosting the accuracy of sentiment analysis in comparison to the fixed-length processing method of dividing or filling samples.

Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning

Image captioning is oriented towards describing an image with the best possible use of words that can provide a semantic, relatable meaning of the scenario inscribed. Different models can be used to accomplish this arduous task depending on the context and requirement of what needs to be achieved. An encoder–decoder model which uses the image feature vectors as an input to the encoder is often marked as one of the appropriate models to accomplish the captioning process. In the proposed work, a dual-modal transformer has been used which captures the intra- and inter-model interactions in a simultaneous manner within an attention block. The transformer architecture is quantitatively evaluated on a publicly available Microsoft Common Objects in Context (MS COCO) dataset yielding a Bilingual Evaluation Understudy (BLEU)-4 Score of 85.01. The efficacy of the model is evaluated on Flickr 8k, Flickr 30k datasets and MS COCO datasets and results for the same is compared and analysed with the state-of-the-art methods. The results shows that the proposed model outperformed when compared with conventional models, such as the encoder–decoder model and attention model.

Extracting Effective Image Attributes with Refined Universal Detection

Recently, image attributes containing high-level semantic information have been widely used in computer vision tasks, including visual recognition and image captioning. Existing attribute extraction methods map visual concepts to the probabilities of frequently-used words by directly using Convolutional Neural Networks (CNNs). Typically, two main problems exist in those methods. First, words of different parts of speech (POSs) are handled in the same way, but non-nominal words can hardly be mapped to visual regions through CNNs only. Second, synonymous nominal words are treated as independent and different words, in which similarities are ignored. In this paper, a novel Refined Universal Detection (RUDet) method is proposed to solve these two problems. Specifically, a Refinement (RF) module is designed to extract refined attributes of non-nominal words based on the attributes of nominal words and visual features. In addition, a Word Tree (WT) module is constructed to integrate synonymous nouns, which ensures that similar words hold similar and more accurate probabilities. Moreover, a Feature Enhancement (FE) module is adopted to enhance the ability to mine different visual concepts in different scales. Experiments conducted on the large-scale Microsoft (MS) COCO dataset illustrate the effectiveness of our proposed method.

A Survey on Deep Learning in Image Polarity Detection: Balancing Generalization Performances and Computational Costs

Deep convolutional neural networks (CNNs) provide an effective tool to extract complex information from images. In the area of image polarity detection, CNNs are customarily utilized in combination with transfer learning techniques to tackle a major problem: the unavailability of large sets of labeled data. Thus, polarity predictors in general exploit a pre-trained CNN as the feature extractor that in turn feeds a classification unit. While the latter unit is trained from scratch, the pre-trained CNN is subject to fine-tuning. As a result, the specific CNN architecture employed as the feature extractor strongly affects the overall performance of the model. This paper analyses state-of-the-art literature on image polarity detection and identifies the most reliable CNN architectures. Moreover, the paper provides an experimental protocol that should allow assessing the role played by the baseline architecture in the polarity detection task. Performance is evaluated in terms of both generalization abilities and computational complexity. The latter attribute becomes critical as polarity predictors, in the era of social networks, might need to be updated within hours or even minutes. In this regard, the paper gives practical hints on the advantages and disadvantages of the examined architectures both in terms of generalization and computational cost.