Sensing Urban Transportation Events from Multi-Channel Social Signals with the Word2vec Fusion Model

ConAnomaly: Content-Based Anomaly Detection for System Logs

Enterprise systems typically produce a large number of logs to record runtime states and important events. Log anomaly detection is efficient for business management and system maintenance. Most existing log-based anomaly detection methods use log parser to get log event indexes or event templates and then utilize machine learning methods to detect anomalies. However, these methods cannot handle unknown log types and do not take advantage of the log semantic information. In this article, we propose ConAnomaly, a log-based anomaly detection model composed of a log sequence encoder (log2vec) and multi-layer Long Short Term Memory Network (LSTM). We designed log2vec based on the Word2vec model, which first vectorized the words in the log content, then deleted the invalid words through part of speech tagging, and finally obtained the sequence vector by the weighted average method. In this way, ConAnomaly not only captures semantic information in the log but also leverages log sequential relationships. We evaluate our proposed approach on two log datasets. Our experimental results show that ConAnomaly has good stability and can deal with unseen log types to a certain extent, and it provides better performance than most log-based anomaly detection methods.

Traffic accident detection and condition analysis based on social networking data

Accurate detection of traffic accidents as well as condition analysis are essential to effectively restoring traffic flow and reducing serious injuries and fatalities. This goal can be obtained using an advanced data classification model with a rich source of traffic information. Several systems based on sensors and social networking platforms have been presented recently to detect traffic events and monitor traffic conditions. However, sensor-based systems provide limited information, and may fail owing to the long detection times and high false-alarm rates. In addition, social networking data are unstructured, unpredictable, and contain idioms, jargon, and dynamic topics. The machine learning algorithms utilized for traffic event detection might not extract valuable information from social networking data. In this paper, a social network-based, real-time monitoring framework is proposed for traffic accident detection and condition analysis using ontology and latent Dirichlet allocation (OLDA) and bidirectional long short-term memory (Bi-LSTM). First, the query-based search engine effectively collects traffic information from social networks, and the data preprocessing module transforms it into structured form. Second, the proposed OLDA-based topic modeling method automatically labels each sentence (e.g., traffic or non-traffic) to identify the exact traffic information. In addition, the ontology-based event recognition approach detects traffic events from traffic-related data. Next, the sentiment analysis technique identifies the polarity of traffic events employing user's opinions, which helps determine accurate conditions of traffic events. Finally, the FastText model and Bi-LSTM with softmax regression are trained for traffic event detection and condition analysis. The proposed framework is evaluated using traffic-related data, comparing OLDA and Bi-LSTM with existing topic modeling methods and traditional classifiers using word embedding models, respectively. Our system outperforms state-of-the-art methods and achieves accuracy of 97 %. This finding demonstrates that the proposed system is more efficient for traffic event detection and condition analysis, in comparison to other existing systems.

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

Log anomaly detection is an efficient method to manage modern large-scale Internet of Things (IoT) systems. More and more works start to apply natural language processing (NLP) methods, and in particular word2vec, in the log feature extraction. Word2vec can extract the relevance between words and vectorize the words. However, the computing cost of training word2vec is high. Anomalies in logs are dependent on not only an individual log message but also on the log message sequence. Therefore, the vector of words from word2vec can not be used directly, which needs to be transformed into the vector of log events and further transformed into the vector of log sequences. To reduce computational cost and avoid multiple transformations, in this paper, we propose an offline feature extraction model, named LogEvent2vec, which takes the log event as input of word2vec to extract the relevance between log events and vectorize log events directly. LogEvent2vec can work with any coordinate transformation methods and anomaly detection models. After getting the log event vector, we transform log event vector to log sequence vector by bary or tf-idf and three kinds of supervised models (Random Forests, Naive Bayes, and Neural Networks) are trained to detect the anomalies. We have conducted extensive experiments on a real public log dataset from BlueGene/L (BGL). The experimental results demonstrate that LogEvent2vec can significantly reduce computational time by 30 times and improve accuracy, comparing with word2vec. LogEvent2vec with bary and Random Forest can achieve the best F1-score and LogEvent2vec with tf-idf and Naive Bayes needs the least computational time.