Locally active globally stable dynamical systems: Theory, learning, and experiments

State-dependent dynamical systems (DSs) offer adaptivity, reactivity, and robustness to perturbations in motion planning and physical human–robot interaction tasks. Learning DS-based motion plans from non-linear reference trajectories is an active research area in robotics. Most approaches focus on learning DSs that can (i) accurately mimic the demonstrated motion, while (ii) ensuring convergence to the target, i.e., they are globally asymptotically (or exponentially) stable. When subject to perturbations, a compliant robot guided with a DS will continue following the next integral curves of the DS towards the target. If the task requires the robot to track a specific reference trajectory, this approach will fail. To alleviate this shortcoming, we propose the locally active globally stable DS (LAGS-DS), a novel DS formulation that provides both global convergence and stiffness-like symmetric attraction behaviors around a reference trajectory in regions of the state space where trajectory tracking is important. This allows for a unified approach towards motion and impedance encoding in a single DS-based motion model, i.e., stiffness is embedded in the DS. To learn LAGS-DS from demonstrations we propose a learning strategy based on Bayesian non-parametric Gaussian mixture models, Gaussian processes, and a sequence of constrained optimization problems that ensure estimation of stable DS parameters via Lyapunov theory. We experimentally validated LAGS-DS on writing tasks with a KUKA LWR 4+ arm and on navigation and co-manipulation tasks with iCub humanoid robots.

Adaptive Grasping of Moving Objects through Tactile Sensing

A robot’s ability to grasp moving objects depends on the availability of real-time sensor data in both the far-field and near-field of the gripper. This research investigates the potential contribution of tactile sensing to a task of grasping an object in motion. It was hypothesised that combining tactile sensor data with a reactive grasping strategy could improve its robustness to prediction errors, leading to a better, more adaptive performance. Using a two-finger gripper, we evaluated the performance of two algorithms to grasp a ball rolling on a horizontal plane at a range of speeds and gripper contact points. The first approach involved an adaptive grasping strategy initiated by tactile sensors in the fingers. The second strategy initiated the grasp based on a prediction of the position of the object relative to the gripper, and provided a proxy to a vision-based object tracking system. It was found that the integration of tactile sensor feedback resulted in a higher observed grasp robustness, especially when the gripper–ball contact point was displaced from the centre of the gripper. These findings demonstrate the performance gains that can be attained by incorporating near-field sensor data into the grasp strategy and motivate further research on how this strategy might be expanded for use in different manipulator designs and in more complex grasp scenarios.

Robot Anticipation Learning System for Ball Catching

Catching flying objects is a challenging task in human–robot interaction. Traditional techniques predict the intersection position and time using the information obtained during the free-flying ball motion. A common pain point in these systems is the short ball flight time and uncertainties in the ball’s trajectory estimation. In this paper, we present the Robot Anticipation Learning System (RALS) that accounts for the information obtained from observation of the thrower’s hand motion before the ball is released. RALS takes extra time for the robot to start moving in the direction of the target before the opponent finishes throwing. To the best of our knowledge, this is the first robot control system for ball-catching with anticipation skills. Our results show that the information fused from both throwing and flying motions improves the ball-catching rate by up to 20% compared to the baseline approach, with the predictions relying only on the information acquired during the flight phase.

Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces

In industrial or surgical settings, to achieve many tasks successfully, at least two people are needed. To this end, robotic assistance could be used to enable a single person to perform such tasks alone, with the help of robots through direct, shared, or autonomous control. We are interested in four-arm manipulation scenarios, where both feet are used to control two robotic arms via bi-pedal haptic interfaces. The robotic arms complement the tasks of the biological arms, for instance, in supporting and moving an object while working on it (using both hands). To reduce fatigue, cognitive workload, and to ease the execution of the foot manipulation, we propose two types of assistance that can be enabled upon contact with the object (i.e., based on the interaction forces): autonomous-contact force generation and auto-coordination of the robotic arms. The latter relates to controlling both arms with a single foot, once the object is grasped. We designed four (shared) control strategies that are derived from the combinations (absence/presence) of both assistance modalities, and we compared them through a user study (with 12 participants) on a four-arm manipulation task. The results show that force assistance positively improves human–robot fluency in the four-arm task, the ease of use and usefulness; it also reduces the fatigue. Finally, to make the dual-assistance approach the preferred and most successful among the proposed control strategies, delegating the grasping force to the robotic arms is a crucial factor when controlling them both with a single foot.

Learning Mobile Manipulation through Deep Reinforcement Learning

Mobile manipulation has a broad range of applications in robotics. However, it is usually more challenging than fixed-base manipulation due to the complex coordination of a mobile base and a manipulator. Although recent works have demonstrated that deep reinforcement learning is a powerful technique for fixed-base manipulation tasks, most of them are not applicable to mobile manipulation. This paper investigates how to leverage deep reinforcement learning to tackle whole-body mobile manipulation tasks in unstructured environments using only on-board sensors. A novel mobile manipulation system which integrates the state-of-the-art deep reinforcement learning algorithms with visual perception is proposed. It has an efficient framework decoupling visual perception from the deep reinforcement learning control, which enables its generalization from simulation training to real-world testing. Extensive simulation and experiment results show that the proposed mobile manipulation system is able to grasp different types of objects autonomously in various simulation and real-world scenarios, verifying the effectiveness of the proposed mobile manipulation system.

Batting an in-flight object to the target

Striking a flying object such as a ball to some target location is a highly skillful maneuver that a human being has to learn through a great deal of practice. In robotic manipulation, precision batting remains one of the most challenging tasks in which computer vision, modeling, planning, control, and action must be tightly coordinated in a split second. This paper investigates the problem of a two-degree-of-freedom robotic arm intercepting an object in free flight and redirecting it to some target with a single strike, assuming all the movements take place in one vertical plane. Two-dimensional impact is solved under Coulomb friction and energy-based restitution with a proof of termination. Planning combines impact dynamics and projectile flight mechanics with manipulator kinematics and image-based motion estimation. As the object is on the incoming flight, the post-impact task constraint of reaching the target is propagated backward in time, while the arm’s kinematic constraints are propagated forward (via joint trajectory interpolation), all to the pre-impact instant when they will meet constraints that allow batting to happen. All the constraints (16 in total) are then exerted on the arm’s pre-impact joint angles and velocities, which are repeatedly planned based on updated estimates of the object’s motion captured by a high-speed camera. The arm keeps adjusting its motion in sync with planning until batting takes place. Experiments have demonstrated a better batting performance by a Barrett Technology WAM Arm than by a human being without training.

A unified framework for coordinated multi-arm motion planning

Coordination is essential in the design of dynamic control strategies for multi-arm robotic systems. Given the complexity of the task and dexterity of the system, coordination constraints can emerge from different levels of planning and control. Primarily, one must consider task-space coordination, where the robots must coordinate with each other, with an object or with a target of interest. Coordination is also necessary in joint space, as the robots should avoid self-collisions at any time. We provide such joint-space coordination by introducing a centralized inverse kinematics (IK) solver under self-collision avoidance constraints, formulated as a quadratic program and solved in real-time. The space of free motion is modeled through a sparse non-linear kernel classification method in a data-driven learning approach. Moreover, we provide multi-arm task-space coordination for both synchronous or asynchronous behaviors. We define a synchronous behavior as that in which the robot arms must coordinate with each other and with a moving object such that they reach for it in synchrony. In contrast, an asynchronous behavior allows for each robot to perform independent point-to-point reaching motions. To transition smoothly from asynchronous to synchronous behaviors and vice versa, we introduce the notion of synchronization allocation. We show how this allocation can be controlled through an external variable, such as the location of the object to be manipulated. Both behaviors and their synchronization allocation are encoded in a single dynamical system. We validate our framework on a dual-arm robotic system and demonstrate that the robots can re-synchronize and adapt the motion of each arm while avoiding self-collision within milliseconds. The speed of control is exploited to intercept fast moving objects whose motion cannot be predicted accurately.