Learning from Skeleton Data for Human Action Recognition

Abstract

Human action recognition is a fundamental task in human-centred scene under standing and has achieved much progress in computer vision and multimedia. Re cently, skeleton-based action recognition has become much more popular with the help of inexpensive motion sensors and effective pose estimation algorithms. As the skeleton data typically are light-weight, view-invariant and privacy-friendly to its video counterparts, a wide range of applications, such as human-computer in teractions and healthcare assistance, can therefore benefit from these features. As human body skeleton is naturally constructed as a spatial-temporal graph instead of a sequence of vector or pseudo-image, where the topology information is more informative for action recognition, the recent graph convolutional neural networks (GCNs) are then proposed for learning representations on such graph structured data and have dominated skeleton-based action recognition tasks. In this thesis we focus on human action recognition from skeleton data using GCN-based methods. In my first work, a novel dual-head GCN model is proposed aims to jointly capture fine-grained and coarse-grained motion patterns efficiently. In this dual head network, each head focuses on specific granularity of temporal motions and hence is more effective. In my second work, a novel long short-term feature ag gregation strategy is proposed to model the varied spatial-temporal dependencies, which is also a key to recognise human actions in skeleton sequences. This novel factorised architecture can alternately perform spatial and temporal feature aggre gation. The aforementioned two works focus on the problem of many-shot clas sification, where each class has a substantial amount of samples during training. Nevertheless, the acquisition of well-annotated skeletal sequences is labour-intensive and time-consuming. In my third work, to alleviate the data collection burden, a part-aware prototypical representation learning strategy is proposed for one-shot skeleton-based action recognition. This novel part-aware model captures skeleton motion patterns at global and part levels which is rarely investigated. Extensive ex periments are conducted on public datasets and models achieve the state-of-the-art performance on all of the corresponding benchmarks.