Robust Unsupervised Video Anomaly Detection by Multipath Frame Prediction

SSIM over MSE: A new perspective for video anomaly detection

Video anomaly detection plays a crucial role in ensuring public safety. Its goal is to detect abnormal patterns contained in video frames. Most existing models distinguish the anomalies based on the Mean Squared Error (MSE), which is hard to align with human perception, resulting in discrepancies between model-detected anomalies and those recognized by humans. Unlike the Human Visual System (HVS), those models are trained to prioritize texture over shape, which leads to poor model interpretability and limited performance. To address these limitations, we propose to optimize the video anomaly detection models from the perspective of human visual relevance. The optimization infrastructure includes a novel Structural Similarity Index (SSIM) based loss, a novel anomaly score calculation method based on SSIM, and a spatial-temporal enhancement block in 3D convolution (STE-3D). SSIM loss helps the model emphasize shape information in videos rather than texture. An anomaly score method based on SSIM evaluates video frames to align more closely with human visual perception. STE-3D improves the model's capacity to capture spatial-temporal features and compensates for the deficiency of the SSIM loss in capturing temporal features. STE-3D is lightweight in design and seamlessly integrated into existing video anomaly detection models based on 3D convolution. Extensive experiments and ablation studies were conducted in four challenging video anomaly detection benchmarks,i.e., UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech. The experimental results validate the efficacy of the proposed approaches in improving video anomaly detection performance.

Scene-Dependent Prediction in Latent Space for Video Anomaly Detection and Anticipation

Video anomaly detection (VAD) plays a crucial role in intelligent surveillance. However, an essential type of anomaly named scene-dependent anomaly is overlooked. Moreover, the task of video anomaly anticipation (VAA) also deserves attention. To fill these gaps, we build a comprehensive dataset named NWPU Campus, which is the largest semi-supervised VAD dataset and the first dataset for scene-dependent VAD and VAA. Meanwhile, we introduce a novel forward-backward framework for scene-dependent VAD and VAA, in which the forward network individually solves the VAD and jointly solves the VAA with the backward network. Particularly, we propose a scene-dependent generative model in latent space for the forward and backward networks. First, we propose a hierarchical variational auto-encoder to extract scene-generic features. Next, we design a score-based diffusion model in latent space to refine these features more compact for the task and generate scene-dependent features with a scene information auto-encoder, modeling the relationships between video events and scenes. Finally, we develop a temporal loss from key frames to constrain the motion consistency of video clips. Extensive experiments demonstrate that our method can handle both scene-dependent anomaly detection and anticipation well, achieving state-of-the-art performance on ShanghaiTech, CUHK Avenue, and the proposed NWPU Campus datasets.

Heuristic Optimal Scheduling for Road Traffic Incident Detection Under Computational Constraints

The intelligent monitoring of road surveillance videos is a crucial tool for detecting and predicting traffic anomalies, swiftly identifying road safety risks, rapidly addressing potential hazards, and preventing accidents or secondary incidents. With the vast number of surveillance cameras in operation, conducting traditional real-time video analysis across all cameras at once requires substantial computational resources. Alternatively, methods that employ periodic camera patrol analysis frequently overlook a significant number of anomalous traffic events, thereby hindering the effectiveness of traffic event detection. To overcome these challenges, this paper introduces a heuristic optimal scheduling approach designed to enhance traffic event detection efficiency while operating within limited computational resources. This method leverages historical data and prior knowledge to compute a weighted event feature value for each camera, providing a quantitative measure of its detection efficiency. To optimize resource allocation, a cyclic elimination mechanism is implemented to exclude low-performing cameras, enabling the dynamic reallocation of resources to higher-performing cameras, thereby enhancing overall detection performance. Finally, the effectiveness of the proposed method is validated through a case study conducted in a representative region of a major metropolitan city in China. The results revealed a substantial improvement in traffic event detection efficiency, with increases of 40%, 28%, 17%, and 28% across different time periods when compared to the pre-optimized state. Furthermore, the proposed method outperformed existing resource scheduling algorithms in terms of average load degree, load balance degree, and higher computational resource utilization. By avoiding the common issues of resource wastage and insufficiency often found in static allocation models, this approach offers greater flexibility and adaptability in computational resource scheduling, thereby effectively addressing the practical demands of traffic anomaly detection and early warning systems.

Clustering Aided Weakly Supervised Training to Detect Anomalous Events in Surveillance Videos

Formulating learning systems for the detection of real-world anomalous events using only video-level labels is a challenging task mainly due to the presence of noisy labels as well as the rare occurrence of anomalous events in the training data. We propose a weakly supervised anomaly detection system that has multiple contributions including a random batch selection mechanism to reduce interbatch correlation and a normalcy suppression block (NSB) which learns to minimize anomaly scores over normal regions of a video by utilizing the overall information available in a training batch. In addition, a clustering loss block (CLB) is proposed to mitigate the label noise and to improve the representation learning for the anomalous and normal regions. This block encourages the backbone network to produce two distinct feature clusters representing normal and anomalous events. An extensive analysis of the proposed approach is provided using three popular anomaly detection datasets including UCF-Crime, ShanghaiTech, and UCSD Ped2. The experiments demonstrate the superior anomaly detection capability of our approach.

Distilling Privileged Knowledge for Anomalous Event Detection From Weakly Labeled Videos

Weakly supervised video anomaly detection (WS-VAD) aims to identify the snippets involving anomalous events in long untrimmed videos, with solely text video-level binary labels. A typical paradigm among the existing text WS-VAD methods is to employ multiple modalities as inputs, e.g., RGB, optical flow, and audio, as they can provide sufficient discriminative clues that are robust to the diverse, complicated real-world scenes. However, such a pipeline has high reliance on the availability of multiple modalities and is computationally expensive and storage demanding in processing long sequences, which limits its use in some applications. To address this dilemma, we propose a privileged knowledge distillation (KD) framework dedicated to the WS-VAD task, which can maintain the benefits of exploiting additional modalities, while avoiding the need for using multimodal data in the inference phase. We argue that the performance of the privileged KD framework mainly depends on two factors: 1) the effectiveness of the multimodal teacher network and 2) the completeness of the useful information transfer. To obtain a reliable teacher network, we propose a text cross-modal interactive learning strategy and an anomaly normal discrimination loss, which target learning task-specific cross-modal features and encourage the separability of anomalous and normal representations, respectively. Furthermore, we design both representation- and text logits-level distillation loss functions, which force the unimodal student network to distill abundant privileged knowledge from the text well-trained multimodal teacher network, in a snippet-to-video fashion. Extensive experimental results on three public benchmarks demonstrate that the proposed privileged KD framework can train a lightweight yet effective detector, for localizing anomaly events under the supervision of video-level annotations.

Cross-modality integration framework with prediction, perception and discrimination for video anomaly detection

Video anomaly detection is an important task for public security in the multimedia field. It aims to distinguish events that deviate from normal patterns. As important semantic representation, the textual information can effectively perceive different contents for anomaly detection. However, most existing methods primarily rely on visual modality, with limited incorporation of textual modality in anomaly detection. In this paper, a cross-modality integration framework (CIForAD) is proposed for anomaly detection, which combines both textual and visual modalities for prediction, perception and discrimination. Firstly, a feature fusion prediction (FUP) module is designed to predict the target regions by fusing the visual features and textual features for prompting, which can amplify the discriminative distance. Then an image-text semantic perception (ISP) module is developed to judge semantic consistency by associating the fine-grained visual features with textual features, where a strategy of local training and global inference is introduced to perceive local details and global semantic correlation. Finally, a self-supervised time attention discrimination (TAD) module is built to explore the inter-frame relation and further distinguish abnormal sequences from normal sequences. Extensive experiments on the three challenging benchmarks indicate that our CIForAD obtains state-of-the-art anomaly detection performance.

Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data

Standard machine learning is unable to accommodate inputs which do not belong to the training distribution. The resulting models often give rise to confident incorrect predictions which may lead to devastating consequences. This problem is especially demanding in the context of dense prediction since input images may be only partially anomalous. Previous work has addressed dense out-of-distribution detection by discriminative training with respect to off-the-shelf negative datasets. However, real negative data may lead to over-optimistic evaluation due to possible overlap with test anomalies. To this end, we extend this approach by generating synthetic negative patches along the border of the inlier manifold. We leverage a jointly trained normalizing flow due to a coverage-oriented learning objective and the capability to generate samples at different resolutions. We detect anomalies according to a principled information-theoretic criterion which can be consistently applied through training and inference. The resulting models set the new state of the art on benchmarks for out-of-distribution detection in road-driving scenes and remote sensing imagery despite minimal computational overhead.

Cognitive Refined Augmentation for Video Anomaly Detection in Weak Supervision

Weakly supervised video anomaly detection is a methodology that assesses anomaly levels in individual frames based on labeled video data. Anomaly scores are computed by evaluating the deviation of distances derived from frames in an unbiased state. Weakly supervised video anomaly detection encounters the formidable challenge of false alarms, stemming from various sources, with a major contributor being the inadequate reflection of frame labels during the learning process. Multiple instance learning has been a pivotal solution to this issue in previous studies, necessitating the identification of discernible features between abnormal and normal segments. Simultaneously, it is imperative to identify shared biases within the feature space and cultivate a representative model. In this study, we introduce a novel multiple instance learning framework anchored on a memory unit, which augments features based on memory and effectively bridges the gap between normal and abnormal instances. This augmentation is facilitated through the integration of an multi-head attention feature augmentation module and loss function with a KL divergence and a Gaussian distribution estimation-based approach. The method identifies distinguishable features and secures the inter-instance distance, thus fortifying the distance metrics between abnormal and normal instances approximated by distribution. The contribution of this research involves proposing a novel framework based on MIL for performing WSVAD and presenting an efficient integration strategy during the augmentation process. Extensive experiments were conducted on benchmark datasets XD-Violence and UCF-Crime to substantiate the effectiveness of the proposed model.

Unsupervised Video Anomaly Detection Based on Similarity with Predefined Text Descriptions

Research on video anomaly detection has mainly been based on video data. However, many real-world cases involve users who can conceive potential normal and abnormal situations within the anomaly detection domain. This domain knowledge can be conveniently expressed as text descriptions, such as "walking" or "people fighting", which can be easily obtained, customized for specific applications, and applied to unseen abnormal videos not included in the training dataset. We explore the potential of using these text descriptions with unlabeled video datasets. We use large language models to obtain text descriptions and leverage them to detect abnormal frames by calculating the cosine similarity between the input frame and text descriptions using the CLIP visual language model. To enhance the performance, we refined the CLIP-derived cosine similarity using an unlabeled dataset and the proposed text-conditional similarity, which is a similarity measure between two vectors based on additional learnable parameters and a triplet loss. The proposed method has a simple training and inference process that avoids the computationally intensive analyses of optical flow or multiple frames. The experimental results demonstrate that the proposed method outperforms unsupervised methods by showing 8% and 13% better AUC scores for the ShanghaiTech and UCFcrime datasets, respectively. Although the proposed method shows -6% and -5% than weakly supervised methods for those datasets, in abnormal videos, the proposed method shows 17% and 5% better AUC scores, which means that the proposed method shows comparable results with weakly supervised methods that require resource-intensive dataset labeling. These outcomes validate the potential of using text descriptions in unsupervised video anomaly detection.

Suspicious Behavior Detection with Temporal Feature Extraction and Time-Series Classification for Shoplifting Crime Prevention

The rise in crime rates in many parts of the world, coupled with advancements in computer vision, has increased the need for automated crime detection services. To address this issue, we propose a new approach for detecting suspicious behavior as a means of preventing shoplifting. Existing methods are based on the use of convolutional neural networks that rely on extracting spatial features from pixel values. In contrast, our proposed method employs object detection based on YOLOv5 with Deep Sort to track people through a video, using the resulting bounding box coordinates as temporal features. The extracted temporal features are then modeled as a time-series classification problem. The proposed method was tested on the popular UCF Crime dataset, and benchmarked against the current state-of-the-art robust temporal feature magnitude (RTFM) method, which relies on the Inflated 3D ConvNet (I3D) preprocessing method. Our results demonstrate an impressive 8.45-fold increase in detection inference speed compared to the state-of-the-art RTFM, along with an F1 score of 92%,outperforming RTFM by 3%. Furthermore, our method achieved these results without requiring expensive data augmentation or image feature extraction.

Automated Road Defect and Anomaly Detection for Traffic Safety: A Systematic Review

Recently, there has been a substantial increase in the development of sensor technology. As enabling factors, computer vision (CV) combined with sensor technology have made progress in applications intended to mitigate high rates of fatalities and the costs of traffic-related injuries. Although past surveys and applications of CV have focused on subareas of road hazards, there is yet to be one comprehensive and evidence-based systematic review that investigates CV applications for Automated Road Defect and Anomaly Detection (ARDAD). To present ARDAD's state-of-the-art, this systematic review is focused on determining the research gaps, challenges, and future implications from selected papers (N = 116) between 2000 and 2023, relying primarily on Scopus and Litmaps services. The survey presents a selection of artefacts, including the most popular open-access datasets (D = 18), research and technology trends that with reported performance can help accelerate the application of rapidly advancing sensor technology in ARDAD and CV. The produced survey artefacts can assist the scientific community in further improving traffic conditions and safety.

Computer Vision Applications in Intelligent Transportation Systems: A Survey

As technology continues to develop, computer vision (CV) applications are becoming increasingly widespread in the intelligent transportation systems (ITS) context. These applications are developed to improve the efficiency of transportation systems, increase their level of intelligence, and enhance traffic safety. Advances in CV play an important role in solving problems in the fields of traffic monitoring and control, incident detection and management, road usage pricing, and road condition monitoring, among many others, by providing more effective methods. This survey examines CV applications in the literature, the machine learning and deep learning methods used in ITS applications, the applicability of computer vision applications in ITS contexts, the advantages these technologies offer and the difficulties they present, and future research areas and trends, with the goal of increasing the effectiveness, efficiency, and safety level of ITS. The present review, which brings together research from various sources, aims to show how computer vision techniques can help transportation systems to become smarter by presenting a holistic picture of the literature on different CV applications in the ITS context.

Research on Anomaly Detection of Surveillance Video Based on Branch-Fusion Net and CSAM

As the monitor probes are used more and more widely these days, the task of detecting abnormal behaviors in surveillance videos has gained widespread attention. The generalization ability and parameter overhead of the model affect how accurate the detection result is. To deal with the poor generalization ability and high parameter overhead of the model in existing anomaly detection methods, we propose a three-dimensional multi-branch convolutional fusion network, named "Branch-Fusion Net". The network is designed with a multi-branch structure not only to significantly reduce parameter overhead but also to improve the generalization ability by understanding the input feature map from different perspectives. To ignore useless features during the model training, we propose a simple yet effective Channel Spatial Attention Module (CSAM), which sequentially focuses attention on key channels and spatial feature regions to suppress useless features and enhance important features. We combine the Branch-Fusion Net and the CSAM as a local feature extraction network and use the Bi-Directional Gated Recurrent Unit (Bi-GRU) to extract global feature information. The experiments are validated on a self-built Crimes-mini dataset, and the accuracy of anomaly detection in surveillance videos reaches 93.55% on the test set. The result shows that the model proposed in the paper significantly improves the accuracy of anomaly detection in surveillance videos with low parameter overhead.

Spectrum Anomaly Detection Based on Spatio-Temporal Network Prediction

With the miniaturization of communication devices, the number of distributed electromagnetic devices is increasing. In order to achieve effective management of the electromagnetic spectrum, prediction and anomaly detection of the spectrum has become increasingly critical. This paper proposes an algorithmic framework for detecting spectrum anomalies using deep learning techniques. More specifically, the framework includes spectrum prediction and anomaly detection. We use the sliding window method to divide the time series, construct multi-timescale historical data, and train the model with normal data to have high accuracy spectrum prediction capability. We analyze and determine the discriminant function to distinguish the spectral anomalies by calculating the differences between the predicted and real data. The experimental results show that the proposed method outperforms existing baseline algorithms based on real-world spectrum measurement data and simulated anomaly data.