Incorporating multi-path risk assessment in transformer-based pedestrian crossing action prediction

This paper proposes a Transformer-based framework for predicting pedestrian crossing actions that uses visualized pedestrian-vehicle collision risks, which are assessed from multiple potential paths. Our framework contains two sequential steps: (1) multi-path risks of a pedestrian-vehicle interaction (PVIs) at each time point are estimated and encoded into an RGB image, which captures a high-density array of safety information. (2) a multi-modal fusion architecture that incorporates both risk images and historical sequential data (e.g., pedestrian action and vehicle velocity) is developed based on the Cross-Attention Transformer. The model outputs are also risk-informed, categorized as yielding, risky crossing, and safe crossing. Experiments are conducted on real-world data from the Euro-PVI dataset. Through two-dimensional mapping tests, risk images are validated to have significant spatiotemporal feature differences and transition associations under different PVIs. The Transformer architecture proves to be an effective method for processing multi-path risk images. Prediction accuracy reaches 87.34% for short-term forecasts (0.5 s ahead), maintains stability as the prediction time horizon progressively extends to 2 s, and improves the prediction of abrupt action switches. For further exploration and validation, the risk image data and imaging code are available at www.github.com/Sivan0227/PVI-Risk-Image.

A Survey of Deep Learning-Based Pedestrian Trajectory Prediction: Challenges and Solutions

Pedestrian trajectory prediction is widely used in various applications, such as intelligent transportation systems, autonomous driving, and social robotics. Precisely forecasting surrounding pedestrians' future trajectories can assist intelligent agents in achieving better motion planning. Currently, deep learning-based trajectory prediction methods have demonstrated superior prediction performance to traditional approaches by learning from trajectory data. However, these methods still face many challenges in improving prediction accuracy, efficiency, and reliability. In this survey, we research the main challenges in deep learning-based pedestrian trajectory prediction methods and study this problem and its solutions through literature collection and analysis. Specifically, we first investigate and analyze the existing literature and surveys on pedestrian trajectory prediction. On this basis, we summarize several main challenges faced by deep learning-based pedestrian trajectory prediction, including motion uncertainty, interaction modeling, scene understanding, data-related issues, and the interpretability of prediction models. We then summarize solutions for each challenge. Subsequently, we introduce mainstream trajectory prediction datasets and analyze the state-of-the-art (SOTA) results reported on them. Finally, we discuss potential research prospects in trajectory prediction, aiming to promote the trajectory prediction community.

Adapting lateral stepping control to walk on winding paths

Most often, gait biomechanics is studied during straight-ahead walking. However, real-life walking imposes various lateral maneuvers people must navigate. Such maneuvers challenge people's lateral balance and can induce falls. Determining how people regulate their stepping movements during such complex walking tasks is therefore essential. Here, 24 adults (12F/12M; Age 25.8±3.5yrs) walked on wide or narrow virtual paths that were either straight, slowly-winding, or quickly-winding. From each trial, we computed time series of participants' step widths and their lateral body positions relative to their path. We applied our Goal Equivalent Manifold framework - an analysis of how task-level redundancy impacts motor regulation - to quantify how participants adjusted their step width and lateral position from step to step as they walked on these paths. On the narrower paths, participants walked with narrower steps and less lateral position and step width variability. They did so by correcting step-to-step deviations in lateral position more, while correcting step-to-step deviations in step width less. On the winding paths, participants took both narrower and more variable steps. Interestingly, on slowly-winding paths, participants corrected step-to-step deviations in step width more by correcting step-to-step deviations in lateral position less: i.e., they prioritized maintaining step width over position. Conversely, on quickly-winding paths, participants strongly corrected step-to-step deviations in both step width and lateral position: i.e., they prioritized maintaining both approximately equally, consistent with trying to maximize their maneuverability. These findings have important implications for persons who have elevated fall risk.

Time-Interval-Based Collision Detection for 4WIS Mobile Robots in Human-Shared Indoor Environments

The recent growth in e-commerce has significantly increased the demand for indoor delivery solutions, highlighting challenges in last-mile delivery. This study presents a time-interval-based collision detection method for Four-Wheel Independent Steering (4WIS) mobile robots operating in human-shared indoor environments, where traditional path following algorithms often create unpredictable movements. By integrating kinematic-based robot trajectory calculation with LiDAR-based human detection and Kalman filter-based prediction, our system enables more natural robot-human interactions. Experimental results demonstrate that our parallel driving mode achieves superior human detection performance compared to conventional Ackermann steering, particularly during cornering and high-speed operations. The proposed method's effectiveness is validated through comprehensive experiments in realistic indoor scenarios, showing its potential for improving the efficiency and safety of indoor autonomous navigation systems.

Human motion similarity evaluation based on deep metric learning

In order to eliminate the impact of camera viewpoint factors and human skeleton differences on the action similarity evaluation and to address the issue of human action similarity evaluation under different viewpoints, a method based on deep metric learning is proposed in this article. The method trains an automatic encoder-decoder deep neural network model by means of a homemade synthetic dataset, which maps the 2D human skeletal key point sequence samples extracted from motion videos into three potential low-dimensional dense spaces. Action feature vectors independent of camera viewpoint and human skeleton structure are extracted in the low-dimensional dense spaces, and motion similarity metrics are performed based on these features, thereby effectively eliminating the effects of camera viewpoint and human skeleton size differences on motion similarity evaluation. Specifically, when extracting the action information feature vectors using the automatic encoder-decoder network model, a sliding window method is used to divide the key point sequences of each limb part into sequence patches, and the action information feature vectors independent of the camera viewpoint and skeleton structure are extracted in a smaller time unit, so as to obtain a more refined action similarity evaluation result. In addition, the dynamic time warping (DWT) algorithm is exploited to align the sequence of action information feature vectors temporally, which solves the problem of temporal axis discrepancies in realizing similarity metrics based on action information feature vectors. More accurate and reliable human action similarity evaluation results were achieved by the loss function composed of three components, namely, cross-reconstruction loss, reconstruction loss and triplet loss. Finally, the performance of the algorithm is evaluated in a homemade dataset, and the experimental results show that the method could effectively eliminate the influence of the differences in camera viewpoints and human skeleton sizes on the similarity evaluation of actions, and generate more reliable and closer to the human subjective perception of similarity evaluation results for human actions captured from different viewpoints or with varying skeleton sizes.

Tracking Upper Limb Motion via Wearable Solutions: Systematic Review of Research From 2011 to 2023

BACKGROUND

The development of wearable solutions for tracking upper limb motion has gained research interest over the past decade. This paper provides a systematic review of related research on the type, feasibility, signal processing techniques, and feedback of wearable systems for tracking upper limb motion, mostly in rehabilitation applications, to understand and monitor human movement.

OBJECTIVE

The aim of this article is to investigate how wearables are used to capture upper limb functions, especially related to clinical and rehabilitation applications.

METHODS

A systematic literature search identified 27 relevant studies published in English from 2011 to 2023, across 4 databases: ACM Digital Library, IEEE Xplore, PubMed, and ScienceDirect. We included papers focusing on motion or posture tracking for the upper limbs, wearable devices, feedback given to end users, and systems having clinical or rehabilitation purposes. We excluded papers focusing on exoskeletons, robotics, prosthetics, orthoses, or activity recognition systems; reviews; and books.

RESULTS

The results from this research focus on wearable devices that are designed to monitor upper limb movement. More specifically, studies were divided into 2 distinct categories: clinical motion tracking (15/27, 56%) and rehabilitation (12/27, 44%), involving healthy individuals and patients, with a total of 439 participants. Among the 27 studies, the majority (19/27) used inertial measurement units to track upper limb movement or smart textiles embedded with sensors. These devices were attached to the body with straps (mostly Velcro), providing flexibility and stability. The developed wearable devices positively influenced user motivation through the provided feedback, with visual feedback being the most common owing to the high level of independence provided. Moreover, a variety of signal processing techniques, such as Kalman and Butterworth filters, were applied to ensure data accuracy. However, limitations persist and include sensor positioning, calibration, and battery life, as well as a lack of clinical data on the effectiveness of these systems. The sampling rate of the data collection ranged from 50 Hz to 2000 Hz, which notably affected data quality and battery life. In addition, several findings were inconclusive, and thus, further future research is needed to understand and improve upper limb posture to develop progressive wearable systems.

CONCLUSIONS

This paper offers a comprehensive overview of wearable monitoring systems, with a focus on upper limb motion tracking and rehabilitation. It emphasizes the various types of available solutions; their efficacy, wearability, and feasibility; and proposed processing techniques. Finally, it presents robust findings regarding feedback accuracy derived from experiments and outlines potential future research directions.

Multimodal Trajectory Prediction for Diverse Vehicle Types in Autonomous Driving with Heterogeneous Data and Physical Constraints

The accurate prediction of vehicle behavior is crucial for autonomous driving systems, impacting their safety and efficiency in complex urban environments. To address the challenge of multi-agent trajectory prediction, we propose a novel model integrating multiple input modalities, including historical trajectories, map data, vehicle features, and interaction information. Our approach employs a Conditional Variational Autoencoder (CVAE) framework with a decoder that predicts control actions using the Gaussian Mixture Model (GMM) and then converts these actions into dynamically feasible trajectories through a bicycle model. Evaluated on the nuScenes dataset, the model achieves great performance across key metrics, including minADE5 of 1.26 and minFDE5 of 2.85, demonstrating robust performance across various vehicle types and prediction horizons. These results indicate that integrating multiple data sources, physical models, and probabilistic methods significantly improves trajectory prediction accuracy and reliability for autonomous driving. Our approach generates diverse yet realistic predictions, capturing the multimodal nature of future outcomes while adhering to Physical Constraints and vehicle dynamics.

Human-AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era

Remote sighted assistance (RSA) has emerged as a conversational technology aiding people with visual impairments (VI) through real-time video chat communication with sighted agents. We conducted a literature review and interviewed 12 RSA users to understand the technical and navigational challenges faced by both agents and users. The technical challenges were categorized into four groups: agents' difficulties in orienting and localizing users, acquiring and interpreting users' surroundings and obstacles, delivering information specific to user situations, and coping with poor network connections. We also presented 15 real-world navigational challenges, including 8 outdoor and 7 indoor scenarios. Given the spatial and visual nature of these challenges, we identified relevant computer vision problems that could potentially provide solutions. We then formulated 10 emerging problems that neither human agents nor computer vision can fully address alone. For each emerging problem, we discussed solutions grounded in human-AI collaboration. Additionally, with the advent of large language models (LLMs), we outlined how RSA can integrate with LLMs within a human-AI collaborative framework, envisioning the future of visual prosthetics.

Review of the application of neural network approaches in pedestrian dynamics studies

In recent years, artificial intelligence methods have been widely used in the study of pedestrian dynamics and crowd evacuation. Different neural network models have been proposed and tested using publicly available pedestrian datasets. These studies have shown that different neural network models present large performance differences for different crowd scenarios. To help future research select more appropriate models, this article presents a review of the application of neural network methods in pedestrian dynamics studies. The studies are classified into two categories: pedestrian trajectory prediction and pedestrian behavior prediction. Both categories are discussed in detail from a conceptual perspective, as well as from the viewpoints of methodology, measurement, and results. The review found that the mainstream method of pedestrian trajectory prediction is currently the LSTM-based method, which has adequate accuracy for short-term predictions. Furthermore, the deep neural network is the most popular method for pedestrian behavior prediction. This method can emulate the decision-making process in a complex environment, and it has the potential to revolutionize the study of pedestrian dynamics. Overall, it is found that new methods and datasets are still required to systemize the study of pedestrian dynamics and eventually ensure its wide-scale application in industry.

Bridging Requirements, Planning, and Evaluation: A Review of Social Robot Navigation

Navigation lies at the core of social robotics, enabling robots to navigate and interact seamlessly in human environments. The primary focus of human-aware robot navigation is minimizing discomfort among surrounding humans. Our review explores user studies, examining factors that cause human discomfort, to perform the grounding of social robot navigation requirements and to form a taxonomy of elementary necessities that should be implemented by comprehensive algorithms. This survey also discusses human-aware navigation from an algorithmic perspective, reviewing the perception and motion planning methods integral to social navigation. Additionally, the review investigates different types of studies and tools facilitating the evaluation of social robot navigation approaches, namely datasets, simulators, and benchmarks. Our survey also identifies the main challenges of human-aware navigation, highlighting the essential future work perspectives. This work stands out from other review papers, as it not only investigates the variety of methods for implementing human awareness in robot control systems but also classifies the approaches according to the grounded requirements regarded in their objectives.

Correlation between Harris hip score and gait analysis through artificial intelligence pose estimation in patients after total hip arthroplasty

BACKGROUND

Recently, open pose estimation using artificial intelligence (AI) has enabled the analysis of time series of human movements through digital video inputs. Analyzing a person's actual movement as a digitized image would give objectivity in evaluating a person's physical function. In the present study, we investigated the relationship of AI camera-based open pose estimation with Harris Hip Score (HHS) developed for patient-reported outcome (PRO) of hip joint function.

METHOD

HHS evaluation and pose estimation using AI camera were performed for a total of 56 patients after total hip arthroplasty in Gyeongsang National University Hospital. Joint angles and gait parameters were analyzed by extracting joint points from time-series data of the patient's movements. A total of 65 parameters were from raw data of the lower extremity. Principal component analysis (PCA) was used to find main parameters. K-means cluster, X-squared test, Random forest, and mean decrease Gini (MDG) graph were also applied.

RESULTS

The train model showed 75% prediction accuracy and the test model showed 81.8% reality prediction accuracy in Random forest. "Anklerang_max", "kneeankle_diff", and "anklerang_rl" showed the top 3 Gini importance score in the Mean Decrease Gini (MDG) graph.

CONCLUSION

The present study shows that pose estimation data using AI camera is related to HHS by presenting associated gait parameters. In addition, our results suggest that ankle angle associated parameters could be key factors of gait analysis in patients who undergo total hip arthroplasty.

Generalizing stepping concepts to non-straight walking

People rarely walk in straight lines. Instead, we make frequent turns or other maneuvers. Spatiotemporal parameters fundamentally characterize gait. For straight walking, these parameters are well-defined for the task of walking on a straight path. Generalizing these concepts to non-straight walking, however, is not straightforward. People follow non-straight paths imposed by their environment (sidewalk, windy hiking trail, etc.) or choose readily-predictable, stereotypical paths of their own. People actively maintain lateral position to stay on their path and readily adapt their stepping when their path changes. We therefore propose a conceptually coherent convention that defines step lengths and widths relative to predefined walking paths. Our convention simply re-aligns lab-based coordinates to be tangent to a walker's path at the mid-point between the two footsteps that define each step. We hypothesized this would yield results both more correct and more consistent with notions from straight walking. We defined several common non-straight walking tasks: single turns, lateral lane changes, walking on circular paths, and walking on arbitrary curvilinear paths. For each, we simulated idealized step sequences denoting "perfect" performance with known constant step lengths and widths. We compared results to path-independent alternatives. For each, we directly quantified accuracy relative to known true values. Results strongly confirmed our hypothesis. Our convention returned vastly smaller errors and introduced no artificial stepping asymmetries across all tasks. All results for our convention rationally generalized concepts from straight walking. Taking walking paths explicitly into account as important task goals themselves thus resolves conceptual ambiguities of prior approaches.

Modality Exploration, Retrieval and Adaptation for Trajectory Prediction

Predicting future trajectories of dynamic agents is inherently riddled with uncertainty. Given a certain historical observation, there are multiple plausible future movements people can perform. Notably, these possible movements are usually centralized around a few representative motion patterns, e.g. acceleration, deceleration, turning, etc. In this paper, we propose a novel prediction scheme which explores human behavior modality representations from real-world trajectory data to discover such motion patterns and further uses them to aid in trajectory prediction. To explore potential behavior modalities, we introduce a deep feature clustering process on trajectory features and each cluster can represent a type of modality. Intuitively, each modality is naturally a class, and a classification network can be adopted to retrieve highly probable modalities about to happen in the future according to historical observations. On account of a wide variety of cues existing in the observation (e.g. agents' motion states, semantics of the scene, etc.), we further design a gated aggregation module to fuse different types of cues into a unified feature. Finally, an adaptation process is proposed to adapt a certain modality to specific historical observations and generate fine-grained prediction results. Extensive experiments on four widely-used benchmarks show the superiority of our proposed approach.

PRISMA: A novel approach for deriving probabilistic surrogate safety measures for risk evaluation

Surrogate Safety Measures (SSMs) are used to express road safety in terms of the safety risk in traffic conflicts. Typically, SSMs rely on assumptions regarding the future evolution of traffic participant trajectories to generate a measure of risk, restricting their applicability to scenarios where these assumptions are valid. In response to this limitation, we present the novel Probabilistic RISk Measure derivAtion (PRISMA) method. The objective of the PRISMA method is to derive SSMs that can be used to calculate in real time the probability of a specific event (e.g., a crash). The PRISMA method adopts a data-driven approach to predict the possible future traffic participant trajectories, thereby reducing the reliance on specific assumptions regarding these trajectories. Since the PRISMA is not bound to specific assumptions, the PRISMA method offers the ability to derive multiple SSMs for various scenarios. The occurrence probability of the specified event is based on simulations and combined with a regression model, this enables our derived SSMs to make real-time risk estimations. To illustrate the PRISMA method, an SSM is derived for risk evaluation during longitudinal traffic interactions. Since there is no known method to objectively estimate risk from first principles, i.e., there is no known risk ground truth, it is very difficult, if not impossible, to objectively compare the relative merits of two SSMs. Instead, we provide a method for benchmarking our derived SSM with respect to expected risk trends. The application of the benchmarking illustrates that the SSM matches the expected risk trends. Whereas the derived SSM shows the potential of the PRISMA method, future work involves applying the approach for other types of traffic conflicts, such as lateral traffic conflicts or interactions with vulnerable road users.

Shared control versus traded control in driving: a debate around automation pitfalls

A major question in human-automation interaction is whether tasks should be traded or shared between human and automation. This work presents reflections-which have evolved through classroom debates between the authors over the past 10 years-on these two forms of human-automation interaction, with a focus on the automated driving domain. As in the lectures, we start with a historically informed survey of six pitfalls of automation: (1) Loss of situation and mode awareness, (2) Deskilling, (3) Unbalanced mental workload, (4) Behavioural adaptation, (5) Misuse, and (6) Disuse. Next, one of the authors explains why he believes that haptic shared control may remedy the pitfalls. Next, another author rebuts these arguments, arguing that traded control is the most promising way to improve road safety. This article ends with a common ground, explaining that shared and traded control outperform each other at medium and low environmental complexity, respectively. Practitioner summary: Designers of automation systems will have to consider whether humans and automation should perform tasks alternately or simultaneously. The present article provides an in-depth reflection on this dilemma, which may prove insightful and help guide design. Abbreviations: ACC: Adaptive Cruise Control: A system that can automatically maintain a safe distance from the vehicle in front; AEB: Advanced Emergency Braking (also known as Autonomous Emergency Braking): A system that automatically brakes to a full stop in an emergency situation; AES: Automated Evasive Steering: A system that automatically steers the car back into safety in an emergency situation; ISA: Intelligent Speed Adaptation: A system that can limit engine power automatically so that the driving speed does not exceed a safe or allowed speed.

Generalizing Stepping Concepts To Non-Straight Walking

People rarely walk in straight lines. Instead, we make frequent turns or other maneuvers. Spatiotemporal parameters fundamentally characterize gait. For straight walking, these parameters are well-defined for that task of walking on a straight path. Generalizing these concepts to non-straight walking, however, is not straightforward. People also follow non-straight paths imposed by their environment (store aisle, sidewalk, etc.) or choose readily-predictable, stereotypical paths of their own. People actively maintain lateral position to stay on their path and readily adapt their stepping when their path changes. We therefore propose a conceptually coherent convention that defines step lengths and widths relative to known walking paths. Our convention simply re-aligns lab-based coordinates to be tangent to a walker's path at the mid-point between the two footsteps that define each step. We hypothesized this would yield results both more correct and more consistent with notions from straight walking. We defined several common non-straight walking tasks: single turns, lateral lane changes, walking on circular paths, and walking on arbitrary curvilinear paths. For each, we simulated idealized step sequences denoting "perfect" performance with known constant step lengths and widths. We compared results to path- independent alternatives. For each, we directly quantified accuracy relative to known true values. Results strongly confirmed our hypothesis. Our convention returned vastly smaller errors and introduced no artificial stepping asymmetries across all tasks. All results for our convention rationally generalized concepts from straight walking. Taking walking paths explicitly into account as important task goals themselves thus resolves conceptual ambiguities of prior approaches.

Holistic Spatio-Temporal Graph Attention for Trajectory Prediction in Vehicle-Pedestrian Interactions

Ensuring that intelligent vehicles do not cause fatal collisions remains a persistent challenge due to pedestrians' unpredictable movements and behavior. The potential for risky situations or collisions arising from even minor misunderstandings in vehicle-pedestrian interactions is a cause for great concern. Considerable research has been dedicated to the advancement of predictive models for pedestrian behavior through trajectory prediction, as well as the exploration of the intricate dynamics of vehicle-pedestrian interactions. However, it is important to note that these studies have certain limitations. In this paper, we propose a novel graph-based trajectory prediction model for vehicle-pedestrian interactions called Holistic Spatio-Temporal Graph Attention (HSTGA) to address these limitations. HSTGA first extracts vehicle-pedestrian interaction spatial features using a multi-layer perceptron (MLP) sub-network and max pooling. Then, the vehicle-pedestrian interaction features are aggregated with the spatial features of pedestrians and vehicles to be fed into the LSTM. The LSTM is modified to learn the vehicle-pedestrian interactions adaptively. Moreover, HSTGA models temporal interactions using an additional LSTM. Then, it models the spatial interactions among pedestrians and between pedestrians and vehicles using graph attention networks (GATs) to combine the hidden states of the LSTMs. We evaluate the performance of HSTGA on three different scenario datasets, including complex unsignalized roundabouts with no crosswalks and unsignalized intersections. The results show that HSTGA outperforms several state-of-the-art methods in predicting linear, curvilinear, and piece-wise linear trajectories of vehicles and pedestrians. Our approach provides a more comprehensive understanding of social interactions, enabling more accurate trajectory prediction for safe vehicle navigation.

Predictive hierarchical reinforcement learning for path-efficient mapless navigation with moving target

Deep reinforcement learning (DRL) has been proven as a powerful approach for robot navigation over the past few years. DRL-based navigation does not require the pre-construction of a map, instead, high-performance navigation skills can be learned from trial-and-error experiences. However, recent DRL-based approaches mostly focus on a fixed navigation target. It is noted that when navigating to a moving target without maps, the performance of the standard RL structure drops dramatically on both the success rate and path efficiency. To address the mapless navigation problem with moving target, the predictive hierarchical DRL (pH-DRL) framework is proposed by integrating the long-term trajectory prediction to provide a cost-effective solution. In the proposed framework, the lower-level policy of the RL agent learns robot control actions to a specified goal, and the higher-level policy learns to make long-range planning of shorter navigation routes by sufficiently exploiting the predicted trajectories. By means of making decisions over two level of policies, the pH-DRL framework is robust to the unavoidable errors in long-term predictions. With the application of deep deterministic policy gradient (DDPG) for policy optimization, the pH-DDPG algorithm is developed based on the pH-DRL structure. Finally, through comparative experiments on the Gazebo simulator with several variants of the DDPG algorithm, the results demonstrate that the pH-DDPG outperforms other algorithms and achieves a high success rate and efficiency even though the target moves fast and randomly.

LQR Control and Optimization for Trajectory Tracking of Biomimetic Robotic Fish Based on Unreal Engine

A realistic and visible dynamic simulation platform can significantly facilitate research on underwater robots. This paper uses the Unreal Engine to generate a scene that resembles real ocean environments, before building a visual dynamic simulation platform in conjunction with the Air-Sim system. On this basis, the trajectory tracking of a biomimetic robotic fish is simulated and assessed. More specifically, we propose a particle swarm optimization algorithm-based control strategy to optimize the discrete linear quadratic regulator controller for the trajectory tracking problem, as well as tracking and controlling discrete trajectories with misaligned time series through introducing a dynamic time warping algorithm. Simulation analyses of the biomimetic robotic fish following a straight line, a circular curve without mutation, and a four-leaf clover curve with mutation are carried out. The obtained results verify the feasibility and effectiveness of the proposed control strategy.

From SLAM to Situational Awareness: Challenges and Survey

The capability of a mobile robot to efficiently and safely perform complex missions is limited by its knowledge of the environment, namely the situation. Advanced reasoning, decision-making, and execution skills enable an intelligent agent to act autonomously in unknown environments. Situational Awareness (SA) is a fundamental capability of humans that has been deeply studied in various fields, such as psychology, military, aerospace, and education. Nevertheless, it has yet to be considered in robotics, which has focused on single compartmentalized concepts such as sensing, spatial perception, sensor fusion, state estimation, and Simultaneous Localization and Mapping (SLAM). Hence, the present research aims to connect the broad multidisciplinary existing knowledge to pave the way for a complete SA system for mobile robotics that we deem paramount for autonomy. To this aim, we define the principal components to structure a robotic SA and their area of competence. Accordingly, this paper investigates each aspect of SA, surveying the state-of-the-art robotics algorithms that cover them, and discusses their current limitations. Remarkably, essential aspects of SA are still immature since the current algorithmic development restricts their performance to only specific environments. Nevertheless, Artificial Intelligence (AI), particularly Deep Learning (DL), has brought new methods to bridge the gap that maintains these fields apart from the deployment to real-world scenarios. Furthermore, an opportunity has been discovered to interconnect the vastly fragmented space of robotic comprehension algorithms through the mechanism of Situational Graph (S-Graph), a generalization of the well-known scene graph. Therefore, we finally shape our vision for the future of robotic situational awareness by discussing interesting recent research directions.

Can Genetic Algorithms Be Used for Real-Time Obstacle Avoidance for LiDAR-Equipped Mobile Robots?

Despite significant progress in robot hardware, the number of mobile robots deployed in public spaces remains low. One of the challenges hindering a wider deployment is that even if a robot can build a map of the environment, for instance through the use of LiDAR sensors, it also needs to calculate, in real time, a smooth trajectory that avoids both static and mobile obstacles. Considering this scenario, in this paper we investigate whether genetic algorithms can play a role in real-time obstacle avoidance. Historically, the typical use of genetic algorithms was in offline optimization. To investigate whether an online, real-time deployment is possible, we create a family of algorithms called GAVO that combines genetic algorithms with the velocity obstacle model. Through a series of experiments, we show that a carefully chosen chromosome representation and parametrization can achieve real-time performance on the obstacle avoidance problem.

Artificial Intelligence for skeleton-based physical rehabilitation action evaluation: A systematic review

Performing prescribed physical exercises during home-based rehabilitation programs plays an important role in regaining muscle strength and improving balance for people with different physical disabilities. However, patients attending these programs are not able to assess their action performance in the absence of a medical expert. Recently, vision-based sensors have been deployed in the activity monitoring domain. They are capable of capturing accurate skeleton data. Furthermore, there have been significant advancements in Computer Vision (CV) and Deep Learning (DL) methodologies. These factors have promoted the solutions for designing automatic patient's activity monitoring models. Then, improving such systems' performance to assist patients and physiotherapists has attracted wide interest of the research community. This paper provides a comprehensive and up-to-date literature review on different stages of skeleton data acquisition processes for the aim of physio exercise monitoring. Then, the previously reported Artificial Intelligence (AI) - based methodologies for skeleton data analysis will be reviewed. In particular, feature learning from skeleton data, evaluation, and feedback generation for the purpose of rehabilitation monitoring will be studied. Furthermore, the associated challenges to these processes will be reviewed. Finally, the paper puts forward several suggestions for future research directions in this area.

Socialistic 3D tracking of humans from a mobile robot for a ‘human following robot’ behaviour

Robotic guides take visitors on a tour of a facility. Such robots must always know the position of the visitor for decision-making. Current tracking algorithms largely assume that the person will be nearly always visible. In the robotic guide application, a person’s visibility could be often lost for prolonged periods, especially when the robot is circumventing a corner or making a sharp turn. In such cases, a person cannot quickly come behind the limited field of view rear camera. We propose a new algorithm that can track people for prolonged times under such conditions. The algorithm is benefitted from an application-level heuristic that the person will be nearly always following the robot, which can be used to guess the motion. The proposed work uses a Particle Filter with a ‘follow-the-robot’ motion model for tracking. The tracking is performed in 3D using a monocular camera. Unlike approaches in the literature, the proposed work observes from a moving base that is especially challenging since a rotation of the robot can cause a large sudden change in the position of the human in the image plane that the approaches in the literature would filter out. Tracking in 3D can resolve such errors. The proposed approach is tested for three different indoor scenarios. The results showcase that the approach is significantly better than the baselines including tracking in the image and projecting in 3D, tracking using a randomized (non-social) motion model, tracking using a Kalman Filter and using LSTM for trajectory prediction.

SEEM: A Sequence Entropy Energy-Based Model for Pedestrian Trajectory All-Then-One Prediction

Predicting the future trajectories of pedestrians is of increasing importance for many applications such as autonomous driving and social robots. Nevertheless, current trajectory prediction models suffer from limitations such as lack of diversity in candidate trajectories, poor accuracy, and instability. In this paper, we propose a novel Sequence Entropy Energy-based Model named SEEM, which consists of a generator network and an energy network. Within SEEM we optimize the sequence entropy by taking advantage of the local variational inference of f-divergence estimation to maximize the mutual information across the generator in order to cover all modes of the trajectory distribution, thereby ensuring SEEM achieves full diversity in candidate trajectory generation. Then, we introduce a probability distribution clipping mechanism to draw samples towards regions of high probability in the trajectory latent space, while our energy network determines which trajectory is most representative of the ground truth. This dual approach is our so-called all-then-one strategy. Finally, a zero-centered potential energy regularization is proposed to ensure stability and convergence of the training process. Through experiments on both synthetic and public benchmark datasets, SEEM is shown to substantially outperform the current state-of-the-art approaches in terms of diversity, accuracy and stability of pedestrian trajectory prediction.

Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction

Predicting human motion from historical pose sequence is crucial for a machine to succeed in intelligent interactions with humans. One aspect that has been obviated so far, is the fact that how we represent the skeletal pose has a critical impact on the prediction results. Yet there is no effort that investigates across different pose representation schemes. We conduct an indepth study on various pose representations with a focus on their effects on the motion prediction task. Moreover, recent approaches build upon off-the-shelf RNN units for motion prediction. These approaches process input pose sequence sequentially and inherently have difficulties in capturing long-term dependencies. In this paper, we propose a novel RNN architecture termed AHMR (Attentive Hierarchical Motion Recurrent network) for motion prediction which simultaneously models local motion contexts and a global context. We further explore a geodesic loss and a forward kinematics loss for the motion prediction task, which have more geometric significance than the widely employed L2 loss. Interestingly, we applied our method to a range of articulate objects including human, fish, and mouse. Empirical results show that our approach outperforms the state-of-the-art methods in short-term prediction and achieves much enhanced long-term prediction proficiency, such as retaining natural human-like motions over 50 seconds predictions. Our codes are released.

Online Tracking of Maneuvering Target Trajectory Based on Chaotic Time Series Prediction

Online prediction of maneuvering target trajectory is one of the most popular research directions at present. Specifically, the primary factors balancing, between prediction accuracy and response time, will give the research substance. This paper presents an online trajectory prediction algorithm based on small sample chaotic time series (OTP-SSCT). First, we optimize in terms of data breadth. The dynamic split window is built according to the motion characteristics of the maneuvering target, thus realizing trajectory segmentation and constructing a small sample chaotic time series prediction set. Second, since fully considering the motion patterns of maneuvering targets, we introduce the spatiotemporal features into the particle swarm optimization (PSO) model identification algorithm, which improves the identification sensitivity of key trajectory data points. Furthermore, we propose a feedback optimization strategy of residual compensation to correct the trajectory prediction values to improve the prediction accuracy. For the initial value sensitivity problem of the PSO model identification algorithm, we propose a new initial population strategy, which improves the effectiveness of initial parameters on model identification. Through simulation experiment analysis, it is verified that the proposed OTP-SSCT algorithm achieves better prediction accuracy and faster response time.

Lighting a Path for Autonomous Vehicle Communication: The Effect of Light Projection on the Detection of Reversing Vehicles by Older Adult Pedestrians

Pedestrian understanding of driver intent is key to pedestrian safety on the road and in parking lots. With the development of autonomous vehicles (AVs), the human driver will be removed, and with it, the exchange that occurs between drivers and pedestrians (e.g., head nods, hand gestures). One possible solution for augmenting that communication is an array of high-intensity light-emitting diodes (LEDs) to project vehicle-to-pedestrian (V2P) messages on the ground plane behind a reversing vehicle. This would be particularly beneficial to elderly pedestrians, who are at particular risk of being struck by reversing cars in parking lots. Their downward gaze and slower reaction time make them particularly vulnerable. A survey was conducted to generate designs, and a simulator experiment was conducted to measure detection and reaction times. The study found that elderly pedestrians are significantly more likely to detect an additional projected message on the ground than detect the existing brake light alone when walking in a parking lot.

Benchmarking the utility of maps of dynamics for human-aware motion planning

Robots operating with humans in highly dynamic environments need not only react to moving persons and objects but also to anticipate and adhere to patterns of motion of dynamic agents in their environment. Currently, robotic systems use information about dynamics locally, through tracking and predicting motion within their direct perceptual range. This limits robots to reactive response to observed motion and to short-term predictions in their immediate vicinity. In this paper, we explore how maps of dynamics (MoDs) that provide information about motion patterns outside of the direct perceptual range of the robot can be used in motion planning to improve the behaviour of a robot in a dynamic environment. We formulate cost functions for four MoD representations to be used in any optimizing motion planning framework. Further, to evaluate the performance gain through using MoDs in motion planning, we design objective metrics, and we introduce a simulation framework for rapid benchmarking. We find that planners that utilize MoDs waste less time waiting for pedestrians, compared to planners that use geometric information alone. In particular, planners utilizing both intensity (proportion of observations at a grid cell where a dynamic entity was detected) and direction information have better task execution efficiency.

Stopping by looking: A driver-pedestrian interaction study in a coupled simulator using head-mounted displays with eye-tracking

Automated vehicles (AVs) can perform low-level control tasks but are not always capable of proper decision-making. This paper presents a concept of eye-based maneuver control for AV-pedestrian interaction. Previously, it was unknown whether the AV should conduct a stopping maneuver when the driver looks at the pedestrian or looks away from the pedestrian. A two-agent experiment was conducted using two head-mounted displays with integrated eye-tracking. Seventeen pairs of participants (pedestrian and driver) each interacted in a road crossing scenario. The pedestrians' task was to hold a button when they felt safe to cross the road, and the drivers' task was to direct their gaze according to instructions. Participants completed three 16-trial blocks: (1) Baseline, in which the AV was pre-programmed to yield or not yield, (2) Look to Yield (LTY), in which the AV yielded when the driver looked at the pedestrian, and (3) Look Away to Yield (LATY), in which the AV yielded when the driver did not look at the pedestrian. The driver's eye movements in the LTY and LATY conditions were visualized using a virtual light beam. Crossing performance was assessed based on whether the pedestrian held the button when the AV yielded and released the button when the AV did not yield. Furthermore, the pedestrians' and drivers' acceptance of the mappings was measured through a questionnaire. The results showed that the LTY and LATY mappings yielded better crossing performance than Baseline. Furthermore, the LTY condition was best accepted by drivers and pedestrians. Eye-tracking analyses indicated that the LTY and LATY mappings attracted the pedestrian's attention, while pedestrians still distributed their attention between the AV and a second vehicle approaching from the other direction. In conclusion, LTY control may be a promising means of AV control at intersections before full automation is technologically feasible.

Real-Time Short-Term Pedestrian Trajectory Prediction Based on Gait Biomechanics

The short-term prediction of a person’s trajectory during normal walking becomes necessary in many environments shared by humans and robots. Physics-based approaches based on Newton’s laws of motion seem best suited for short-term predictions, but the intrinsic properties of human walking conflict with the foundations of the basic kinematical models compromising their performance. In this paper, we propose a short-time prediction method based on gait biomechanics for real-time applications. This method relays on a single biomechanical variable, and it has a low computational burden, turning it into a feasible solution to implement in low-cost portable devices. We evaluate its performance from an experimental benchmark where several subjects walked steadily over straight and curved paths. With this approach, the results indicate a performance good enough to be applicable to a wide range of human–robot interaction applications.

Questioning the Anisotropy of Pedestrian Dynamics: An Empirical Analysis with Artificial Neural Networks

Identifying the factors that control the dynamics of pedestrians is a crucial step towards modeling and building various pedestrian-oriented simulation systems. In this article, we empirically explore the influential factors that control the single-file movement of pedestrians and their impact. Our goal in this context is to apply feed-forward neural networks to predict and understand the individual speeds for different densities of pedestrians. With artificial neural networks, we can approximate the fitting function that describes pedestrians’ movement without having modeling bias. Our analysis is focused on the distances and range of interactions across neighboring pedestrians. As indicated by previous research, we find that the speed of pedestrians depends on the distance to the predecessor. Yet, in contrast to classical purely anisotropic approaches—which are based on vision fields and assume that the interaction mainly depends on the distance in front—our results demonstrate that the distance to the follower also significantly influences movement. Using the distance to the follower combined with the subject pedestrian’s headway distance to predict the speed improves the estimation by 18% compared to the prediction using the space in front alone.

Toward Benchmarking of Long-Term Spatio-Temporal Maps of Pedestrian Flows for Human-Aware Navigation

Despite the advances in mobile robotics, the introduction of autonomous robots in human-populated environments is rather slow. One of the fundamental reasons is the acceptance of robots by people directly affected by a robot’s presence. Understanding human behavior and dynamics is essential for planning when and how robots should traverse busy environments without disrupting people’s natural motion and causing irritation. Research has exploited various techniques to build spatio-temporal representations of people’s presence and flows and compared their applicability to plan optimal paths in the future. Many comparisons of how dynamic map-building techniques show how one method compares on a dataset versus another, but without consistent datasets and high-quality comparison metrics, it is difficult to assess how these various methods compare as a whole and in specific tasks. This article proposes a methodology for creating high-quality criteria with interpretable results for comparing long-term spatio-temporal representations for human-aware path planning and human-aware navigation scheduling. Two criteria derived from the methodology are then applied to compare the representations built by the techniques found in the literature. The approaches are compared on a real-world, long-term dataset, and the conception is validated in a field experiment on a robotic platform deployed in a human-populated environment. Our results indicate that continuous spatio-temporal methods independently modeling spatial and temporal phenomena outperformed other modeling approaches. Our results provide a baseline for future work to compare a wide range of methods employed for long-term navigation and provide researchers with an understanding of how these various methods compare in various scenarios.

Social Momentum: Design and Evaluation of a Framework for Socially Competent Robot Navigation

Mobile robots struggle to integrate seamlessly in crowded environments such as pedestrian scenes, often disrupting human activity. One obstacle preventing their smooth integration is our limited understanding of how humans may perceive and react to robot motion. Motivated by recent studies highlighting the benefits of intent-expressive motion for robots operating close to humans, we describe Social Momentum (SM), a planning framework for legible robot motion generation in multiagent domains. We investigate the properties of motion generated by SM via two large-scale user studies: an online, video-based study ( N = 180) focusing on the legibility of motion produced by SM and a lab study ( N = 105) focusing on the perceptions of users navigating next to a robot running SM in a crowded space. Through statistical and thematic analyses of collected data, we present evidence suggesting that (a) motion generated by SM enables quick inference of the robot’s navigation strategy; (b) humans navigating close to a robot running SM follow comfortable, low-acceleration paths; and (c) robot motion generated by SM is positively perceived and indistinguishable from a teleoperated baseline. Through the discussion of experimental insights and lessons learned, this article aspires to inform future algorithmic and experimental design for social robot navigation.

Recursive Least Squares Based Refinement Network for Vehicle Trajectory Prediction

Trajectory prediction of surrounding objects plays a pivotal role in the field of autonomous driving vehicles. In the current rollout process, it suffers from an accumulation of errors, which has a negative impact on prediction accuracy. This paper proposes a parametric-learning recursive least-squares (RLS) method integrated with an encoder–decoder framework for trajectory prediction, named the recursive least-squares-based refinement network (RRN). Through the generation of several anchors in the future trajectory, RRN can capture both local and global motion patterns. We conducted experiments on the prevalent NGSIM and INTERACTION datasets, which contain various scenarios such as highways, intersections and roundabouts. The promising results indicate that RRN could improve the performance of the rollout trajectory prediction effectively.

Multiagent off-screen behavior prediction in football

In multiagent worlds, several decision-making individuals interact while adhering to the dynamics constraints imposed by the environment. These interactions, combined with the potential stochasticity of the agents’ dynamic behaviors, make such systems complex and interesting to study from a decision-making perspective. Significant research has been conducted on learning models for forward-direction estimation of agent behaviors, for example, pedestrian predictions used for collision-avoidance in self-driving cars. In many settings, only sporadic observations of agents may be available in a given trajectory sequence. In football, subsets of players may come in and out of view of broadcast video footage, while unobserved players continue to interact off-screen. In this paper, we study the problem of multiagent time-series imputation in the context of human football play, where available past and future observations of subsets of agents are used to estimate missing observations for other agents. Our approach, called the Graph Imputer, uses past and future information in combination with graph networks and variational autoencoders to enable learning of a distribution of imputed trajectories. We demonstrate our approach on multiagent settings involving players that are partially-observable, using the Graph Imputer to predict the behaviors of off-screen players. To quantitatively evaluate the approach, we conduct experiments on football matches with ground truth trajectory data, using a camera module to simulate the off-screen player state estimation setting. We subsequently use our approach for downstream football analytics under partial observability using the well-established framework of pitch control, which traditionally relies on fully observed data. We illustrate that our method outperforms several state-of-the-art approaches, including those hand-crafted for football, across all considered metrics.

Opportunities for Human-AI Collaboration in Remote Sighted Assistance

Remote sighted assistance (RSA) has emerged as a conversational assistive technology for people with visual impairments (VI), where remote sighted agents provide realtime navigational assistance to users with visual impairments via video-chat-like communication. In this paper, we conducted a literature review and interviewed 12 RSA users to comprehensively understand technical and navigational challenges in RSA for both the agents and users. Technical challenges are organized into four categories: agents' difficulties in orienting and localizing the users; acquiring the users' surroundings and detecting obstacles; delivering information and understanding user-specific situations; and coping with a poor network connection. Navigational challenges are presented in 15 real-world scenarios (8 outdoor, 7 indoor) for the users. Prior work indicates that computer vision (CV) technologies, especially interactive 3D maps and realtime localization, can address a subset of these challenges. However, we argue that addressing the full spectrum of these challenges warrants new development in Human-CV collaboration, which we formalize as five emerging problems: making object recognition and obstacle avoidance algorithms blind-aware; localizing users under poor networks; recognizing digital content on LCD screens; recognizing texts on irregular surfaces; and predicting the trajectory of out-of-frame pedestrians or objects. Addressing these problems can advance computer vision research and usher into the next generation of RSA service.

From Perception to Navigation in Environments with Persons: An Indoor Evaluation of the State of the Art

Research in the field of social robotics is allowing service robots to operate in environments with people. In the aim of realizing the vision of humans and robots coexisting in the same environment, several solutions have been proposed to (1) perceive persons and objects in the immediate environment; (2) predict the movements of humans; as well as (3) plan the navigation in agreement with socially accepted rules. In this work, we discuss the different aspects related to social navigation in the context of our experience in an indoor environment. We describe state-of-the-art approaches and experiment with existing methods to analyze their performance in practice. From this study, we gather first-hand insights into the limitations of current solutions and identify possible research directions to address the open challenges. In particular, this paper focuses on topics related to perception at the hardware and application levels, including 2D and 3D sensors, geometric and mainly semantic mapping, the prediction of people trajectories (physics-, pattern- and planning-based), and social navigation (reactive and predictive) in indoor environments.

A Review on Scene Prediction for Automated Driving

Towards the aim of mastering level 5, a fully automated vehicle needs to be equipped with sensors for a 360∘ surround perception of the environment. In addition to this, it is required to anticipate plausible evolutions of the traffic scene such that it is possible to act in time, not just to react in case of emergencies. This way, a safe and smooth driving experience can be guaranteed. The complex spatio-temporal dependencies and high dynamics are some of the biggest challenges for scene prediction. The subtile indications of other drivers’ intentions, which are often intuitively clear to the human driver, require data-driven models such as deep learning techniques. When dealing with uncertainties and making decisions based on noisy or sparse data, deep learning models also show a very robust performance. In this survey, a detailed overview of scene prediction models is presented with a historical approach. A quantitative comparison of the model results reveals the dominance of deep learning methods in current state-of-the-art research in this area, leading to a competition on the cm scale. Moreover, it also shows the problem of inter-model comparison, as many publications do not use standardized test sets. However, it is questionable if such improvements on the cm scale are actually necessary. More effort should be spent in trying to understand varying model performances, identifying if the difference is in the datasets (many simple situations versus many corner cases) or actually an issue of the model itself.