Swarm intelligence for detecting interesting events in crowded environments

An informative dual ForkNet for video anomaly detection

An autoencoder for video anomaly detection task is a type of algorithm with the primary purpose of learning an "informative" representation of the normal data that can be used for identifying the abnormal data by learning to reconstruct a set of input observations. Based on the encoding-decoding structure, we explore a novel dual ForkNet architecture that can dissociate and process the spatio-temporal representation. It is well-known in the information theory community that most autoencoders coding processes are inevitably accompanied by a certain loss of information. In this dual ForkNet, we focus on mitigating the information loss problem and propose a novel architectural recalibration approach, which we term the "Informetrics Recalibration" (IR). It can adaptively recalibrate latent feature representation by explicitly modeling the similarity between the corresponding feature maps of encoder and decoder, and retain more useful semantic information to generate greater differentiation between normal and abnormal events. Additionally, because the structure of the autoencoder itself determines the difficulty to obtain deep semantic information, we introduce a Secondary Encoder (SE) in each ForkNet, so as to recalibrate target features responses of latent feature representation. Our model is easy to be trained and robust to be applied, because it basically consists of some ResNet blocks without using complicated modules. Extensive experiments on the five publicly available benchmarks show that our model outperforms the existing state-of-the-art architectures, demonstrating our framework's effectiveness.

Audio-visual multi-modality driven hybrid feature learning model for crowd analysis and classification

The high pace emergence in advanced software systems, low-cost hardware and decentralized cloud computing technologies have broadened the horizon for vision-based surveillance, monitoring and control. However, complex and inferior feature learning over visual artefacts or video streams, especially under extreme conditions confine majority of the at-hand vision-based crowd analysis and classification systems. Retrieving event-sensitive or crowd-type sensitive spatio-temporal features for the different crowd types under extreme conditions is a highly complex task. Consequently, it results in lower accuracy and hence low reliability that confines existing methods for real-time crowd analysis. Despite numerous efforts in vision-based approaches, the lack of acoustic cues often creates ambiguity in crowd classification. On the other hand, the strategic amalgamation of audio-visual features can enable accurate and reliable crowd analysis and classification. Considering it as motivation, in this research a novel audio-visual multi-modality driven hybrid feature learning model is developed for crowd analysis and classification. In this work, a hybrid feature extraction model was applied to extract deep spatio-temporal features by using Gray-Level Co-occurrence Metrics (GLCM) and AlexNet transferrable learning model. Once extracting the different GLCM features and AlexNet deep features, horizontal concatenation was done to fuse the different feature sets. Similarly, for acoustic feature extraction, the audio samples (from the input video) were processed for static (fixed size) sampling, pre-emphasis, block framing and Hann windowing, followed by acoustic feature extraction like GTCC, GTCC-Delta, GTCC-Delta-Delta, MFCC, Spectral Entropy, Spectral Flux, Spectral Slope and Harmonics to Noise Ratio (HNR). Finally, the extracted audio-visual features were fused to yield a composite multi-modal feature set, which is processed for classification using the random forest ensemble classifier. The multi-class classification yields a crowd-classification accurac12529y of (98.26%), precision (98.89%), sensitivity (94.82%), specificity (95.57%), and F-Measure of 98.84%. The robustness of the proposed multi-modality-based crowd analysis model confirms its suitability towards real-world crowd detection and classification tasks.

A Review of Different Components of the Intelligent Traffic Management System (ITMS)

Traffic congestion is a serious challenge in urban areas. So, to address this challenge, the intelligent traffic management system (ITMS) is used to manage traffic on road networks. Managing traffic helps to focus on environmental impacts as well as emergency situations. However, the ITMS system has many challenges in analyzing scenes of complex traffic. New technologies such as computer vision (CV) and artificial intelligence (AI) are being used to solve these challenges. As a result, these technologies have made a distinct identity in the surveillance industry, particularly when it comes to keeping a constant eye on traffic scenes. There are many vehicle attributes and existing approaches that are being used in the development of ITMS, along with imaging technologies. In this paper, we reviewed the ITMS-based components that describe existing imaging technologies and existing approaches on the basis of their need for developing ITMS. The first component describes the traffic scene and imaging technologies. The second component talks about vehicle attributes and their utilization in existing vehicle-based approaches. The third component explains the vehicle’s behavior on the basis of the second component’s outcome. The fourth component explains how traffic-related applications can assist in the management and monitoring of traffic flow, as well as in the reduction of congestion and the enhancement of road safety. The fifth component describes the different types of ITMS applications. The sixth component discusses the existing methods of traffic signal control systems (TSCSs). Aside from these components, we also discuss existing vehicle-related tools such as simulators that work to create realistic traffic scenes. In the last section named discussion, we discuss the future development of ITMS and draw some conclusions. The main objective of this paper is to discuss the possible solutions to different problems during the development of ITMS in one place, with the help of components that would play an important role for an ITMS developer to achieve the goal of developing efficient ITMS.

An IoT Enable Anomaly Detection System for Smart City Surveillance

Since the advent of visual sensors, smart cities have generated massive surveillance video data, which can be intelligently inspected to detect anomalies. Computer vision-based automated anomaly detection techniques replace human intervention to secure video surveillance applications in place from traditional video surveillance systems that rely on human involvement for anomaly detection, which is tedious and inaccurate. Due to the diverse nature of anomalous events and their complexity, it is however, very challenging to detect them automatically in a real-world scenario. By using Artificial Intelligence of Things (AIoT), this research work presents an efficient and robust framework for detecting anomalies in surveillance large video data. A hybrid model integrating 2D-CNN and ESN are proposed in this research study for smart surveillance, which is an important application of AIoT. The CNN is used as feature extractor from input videos which are then inputted to autoencoder for feature refinement followed by ESN for sequence learning and anomalous events detection. The proposed model is lightweight and implemented over edge devices to ensure their capability and applicability over AIoT environments in a smart city. The proposed model significantly enhanced performance using challenging surveillance datasets compared to other methods.

Detection of multiple anomalous instances in video surveillance systems

Anomalous event recognition has a complicated definition in the complex background due to the sparse occurrence of anomalies. In this paper, we form a framework for classifying multiple anomalies present in video frames that happen in a context such as the sudden moment of people in various directions and anomalous vehicles in the pedestrian park. An attention U-net model on video frames is utilized to create a binary segmented anomalous image that classifies each anomalous object in the video. White pixels indicate the anomaly, and black pixels serve as the background image. For better segmentation, we have assigned a border to every anomalous object in a binary image. Further to distinguish each anomaly a watershed algorithm is utilized that develops multi-level gray image masks for every anomalous class. This forms a multi-class problem, where each anomalous instance is represented by a different gray color level. We use pixel values, Optical Intensity, entropy values, and Gaussian filter with sigma 5, and 7 to form a feature extraction module for training video images along with their multi-instance gray-level masks. Pixel-level localization and identification of unusual items are done using the feature vectors acquired from the feature extraction module and multi-class stack classifier model. The proposed methodology is evaluated on UCSD Ped1, Ped2 and UMN datasets that obtain pixel-level average accuracy results of 81.15%,87.26% and 82.67% respectively.

Anticollision Decision and Control of UAV Swarm Based on Intelligent Cognitive Game

UAV swarm anticollision system is very important to improve the flight safety of the whole swarm formation, while the existing system design methods are still insufficient in realizing autonomous and cooperative anticollision. Based on the cognitive game theory, an intelligent decision-making and control method for UAV swarm anticollision is designed. Firstly, by using the idea of swarm intelligence, basic flight behaviors of UAV swarm are defined as five basic flight rules, such as cohesion, following, self-guidance, dispersion, and alliance. Further, the cognitive security domain of UAV swarm is constructed by setting the overall anticollision rules of the swarm and the anticollision rules of individual members. On this basis, the anticollision problem of UAV swarm is transformed into a game problem involving two parties, and the solution method of decision and control strategy set is proposed. Finally, the stability of anticollision decision and control method is proved through eigenvalue theory. The simulation results show that the method proposed in this paper can effectively realize the autonomous cooperative anticollision of UAV swarm and also has good algorithm real-time solution ability while ensuring flight safety.

Anomaly detection using edge computing in video surveillance system: review

The current concept of smart cities influences urban planners and researchers to provide modern, secured and sustainable infrastructure and gives a decent quality of life to its residents. To fulfill this need, video surveillance cameras have been deployed to enhance the safety and well-being of the citizens. Despite technical developments in modern science, abnormal event detection in surveillance video systems is challenging and requires exhaustive human efforts. In this paper, we focus on evolution of anomaly detection followed by survey of various methodologies developed to detect anomalies in intelligent video surveillance. Further, we revisit the surveys on anomaly detection in the last decade. We then present a systematic categorization of methodologies for anomaly detection. As the notion of anomaly depends on context, we identify different objects-of-interest and publicly available datasets in anomaly detection. Since anomaly detection is a time-critical application of computer vision, we explore the anomaly detection using edge devices and approaches explicitly designed for them. The confluence of edge computing and anomaly detection for real-time and intelligent surveillance applications is also explored. Further, we discuss the challenges and opportunities involved in anomaly detection using the edge devices.

Deep Vigilante: A deep learning network for real-world crime detection

 Identification/recognition of assault, fighting, shooting, and vandalism from video sequence using deep 2D and 3D convolutional neural networks (CNNs) is explored in this paper. Recent wave of extensive unrestricted urbanization has not only uplifted the standard of living, but has also threatened the safety of a common man leading to an extraordinary rise in crime rate. Although Closed-circuit television (CCTV) footage provides a monitoring framework, yet, it’s useless without an auto volume crime detection system. The system proposed in this work is an effort to eradicate volume crimes through accurate detection in real-time. Firstly, a fine-grained annotated dataset including instance and activity information has been developed for real-world volume crimes. Secondly, a comparison between 3D CNN and 2D CNN network has been presented to identify the malicious event from the video sequence. This is carried out to explore the significance of spatial and temporal information present in the video for event recognition. It has been observed that 2D CNN even with lesser parameters achieved a promising classification accuracy of 91.2%and Area under the curve (AUC) of 95.2%on four classes. The system also reduces false alarm rate in comparison to state-of-the-art approaches.

Trajectory Outlier Detection

Detecting abnormal trajectories is an important task in research and industrial applications, which has attracted considerable attention in recent decades. This work studies the existing trajectory outlier detection algorithms in different industrial domains and applications, including maritime, smart urban transportation, video surveillance, and climate change domains. First, we review several algorithms for trajectory outlier detection. Second, different taxonomies are proposed regarding application-, output-, and algorithm-based levels. Third, evaluation of 10 trajectory outlier detection algorithms is performed on small, large, and big trajectory databases. Finally, future challenges and open issues with regard to trajectory outliers are derived and discussed. This survey offers a general overview of existing trajectory outlier detection algorithms in industrial informatics applications. As a result, mature solutions may be further developed by data mining and machine learning communities.

Advances and Trends in Real Time Visual Crowd Analysis

Real time crowd analysis represents an active area of research within the computer vision community in general and scene analysis in particular. Over the last 10 years, various methods for crowd management in real time scenario have received immense attention due to large scale applications in people counting, public events management, disaster management, safety monitoring an so on. Although many sophisticated algorithms have been developed to address the task; crowd management in real time conditions is still a challenging problem being completely solved, particularly in wild and unconstrained conditions. In the proposed paper, we present a detailed review of crowd analysis and management, focusing on state-of-the-art methods for both controlled and unconstrained conditions. The paper illustrates both the advantages and disadvantages of state-of-the-art methods. The methods presented comprise the seminal research works on crowd management, and monitoring and then culminating state-of-the-art methods of the newly introduced deep learning methods. Comparison of the previous methods is presented, with a detailed discussion of the direction for future research work. We believe this review article will contribute to various application domains and will also augment the knowledge of the crowd analysis within the research community.

BMAN: Bidirectional Multi-scale Aggregation Networks for Abnormal Event Detection

Abnormal event detection is an important task in video surveillance systems. In this paper, we propose a novel bidirectional multi-scale aggregation networks (BMAN) for abnormal event detection. The proposed BMAN learns spatiotemporal patterns of normal events to detect deviations from the learned normal patterns as abnormalities. The BMAN consists of two main parts: an inter-frame predictor and an appearancemotion joint detector. The inter-frame predictor is devised to encode normal patterns, which generates an inter-frame using bidirectional multi-scale aggregation based on attention. With the feature aggregation, robustness for object scale variations and complex motions is achieved in normal pattern encoding. Based on the encoded normal patterns, abnormal events are detected by the appearance-motion joint detector in which both appearance and motion characteristics of scenes are considered. Comprehensive experiments are performed, and the results show that the proposed method outperforms the existing state-of-the-art methods. The resulting abnormal event detection is interpretable on the visual basis of where the detected events occur. Further, we validate the effectiveness of the proposed network designs by conducting ablation study and feature visualization.

Fetal Cardiac Timing Events Estimation From Doppler Ultrasound Signals Using Swarm Decomposition

Perinatal morbidity and mortality can be reduced when any cardiac abnormalities during a pregnancy are diagnosed early. Doppler Ultrasound Signals (DUS) are often used to monitor the heart rate of a fetus and they can also be used to identify the timing events of fetal cardiac valve motions. This paper proposed a novel, non-invasive technique which can be used to identify the fetal cardiac timing events based upon the analysis of fetal DUS (based upon 66 normal subjects belonging to three differing age groups) which can later be used to estimate fetal cardiac intervals from a DUS signal. The foundation of this method is a novel decomposition method referred to as Swarm Decomposition (SWD) which makes it possible for the frequency contents of Doppler signals to be associated with cardiac valve motions. These motions include the opening (o) and closing (c) of Aortic (A) and Mitral (M) valves. When compared the SWD method results to the Empirical Mode Decomposition for the validation, the fetal cardiac timings were estimated successfully when isolating the constituent parts of analyzed DUS signals with reduced complexity compared to EMD method. Pulsed Doppler images are used in order to verify the estimated timings. Three fetal age groups were assessed in terms of their cardiac intervals: 16–29, 30–35, and 36–41 weeks. The time intervals (Systolic Time Interval, STI), (Isovolumic Relaxation Time, IRT), and (Pre-ejection Period, PEP) were found to change significantly (p < 0.05) across the three age groups. The evaluation of fetal cardiac performance can be enhanced, given that these findings can be leveraged as sensitive markers throughout the process.

Variational Bayesian Learning of Generalized Dirichlet-Based Hidden Markov Models Applied to Unusual Events Detection

Learning a hidden Markov model (HMM) is typically based on the computation of a likelihood which is intractable due to a summation over all possible combinations of states and mixture components. This estimation is often tackled by a maximization strategy, which is known as the Baum-Welch algorithm. However, some drawbacks of this approach have led to the consideration of Bayesian methods that add a prior over the parameters in order to work with the posterior probability and the marginal likelihood. These approaches can lead to good models but to the cost of extremely long computations (e.g., Markov Chain Monte Carlo). More recently, variational Bayesian frameworks have been proposed as a Bayesian alternative that keeps the computation tractable and the approximation tight. It relies on the introduction of a prior over the parameters to be learned and on an approximation of the true posterior distribution. After proving good standing in the case of finite mixture models and discrete and Gaussian HMMs, we propose here to derive the equations of the variational learning of the Dirichlet mixture-based HMM, and to extend it to the generalized Dirichlet. The latter case presents several properties that make the estimation more accurate. We prove the validity of this approach within the context of unusual event detection in public areas using the University of California San Diego data sets. HMMs are trained over normal video sequences using the typical Baum-Welch approach versus the variational one. The variational learning leads to more accurate models for the detection and localization of anomaly, and the general HMM approach is shown to be versatile enough to handle the detection of various synthetically generated tampering events.

Gaussian Process Regression-Based Video Anomaly Detection and Localization With Hierarchical Feature Representation

This paper presents a hierarchical framework for detecting local and global anomalies via hierarchical feature representation and Gaussian process regression (GPR) which is fully non-parametric and robust to the noisy training data, and supports sparse features. While most research on anomaly detection has focused more on detecting local anomalies, we are more interested in global anomalies that involve multiple normal events interacting in an unusual manner, such as car accidents. To simultaneously detect local and global anomalies, we cast the extraction of normal interactions from the training videos as a problem of finding the frequent geometric relations of the nearby sparse spatio-temporal interest points (STIPs). A codebook of interaction templates is then constructed and modeled using the GPR, based on which a novel inference method for computing the likelihood of an observed interaction is also developed. Thereafter, these local likelihood scores are integrated into globally consistent anomaly masks, from which anomalies can be succinctly identified. To the best of our knowledge, it is the first time GPR is employed to model the relationship of the nearby STIPs for anomaly detection. Simulations based on four widespread datasets show that the new method outperforms the main state-of-the-art methods with lower computational burden.