Modelling Skeleton-based Human Dynamics via Retrospection

Abstract

Human motion prediction is one of the key problems in computer vision and robotic vision and has received increasing attention in recent years. The target is to generate the future continuous, realistic human poses given a seed sequence, which can further assist human motion analysis. However, due to the high-uncertainty, it is difficult and challenging to model human dynamics which not only requires spatial information including complicated joint correlations, but also temporal information including periodic properties. Recently, deep recurrent neural networks (RNNs) have achieved impressive success in forecasting human motion with a sequence-to-sequence architecture. However, forecasting in longer time horizons often leads to implausible human poses or converges to mean poses, because of error accumulation and difficulties in keeping track of longer-term information. Based on these observations, in this study, we propose to retrospect human dynamics with attention. A retrospection module is designed upon RNN to regularly retrospect past frames and correct mistakes in time. This significantly improves the memory of RNN and provides sufficient information for the decoder networks to generate longer-term predictions. Moreover, we present a spatial attention module to explore cooperation among joints in performing a particular motion as well as a temporal attention module to exploit the level of importance among observed frames. Residual connections are also included to guarantee the performance of short-term prediction. We evaluate the proposed algorithm on the largest and most challenging Human 3.6M dataset in the field. Experimental results demonstrate the necessity of investigating motion prediction in a self-audit manner and the effectiveness of the proposed algorithm in both short-term and long-term predictions.