Signal processing and machine learning techniques for automatic image-based facial expression recognition

Abstract

In this thesis novel signal processing and machine learning techniques are proposed and evaluated for automatic image-based facial expression recognition, which are aimed to progress towards real world operation. A thorough evaluation of the performance of certain image-based expression recognition techniques is performed using a posed database and for the rst time three progressively more challenging spontaneous databases. These methods exploit the principles of sparse representation theory with identity-independent expression recognition using di erence images. The second contribution exploits a low complexity method to extract geometric features from facial expression images. The misalignment problem of the training images is solved and the performance of both geometric and appearance features is assessed on the same three spontaneous databases. A deep network framework that contains auto-encoders is used to form an improved classi er. The nal work focuses upon enhancing the expression recognition performance by the selection and fusion of di erent types of features comprising geometric features and two sorts of appearance features. This provides a rich feature vector by which the best representation of the spontaneous facial features is obtained. Subsequently, the computational complexity is reduced by maintaining important location information by concentrating on the crucial roles of the facial regions as the basic processing instead of the entire face, where the local binary patterns and local phase quantization features are extracted automatically by means of detecting two important regions of the face. Next, an automatic method for splitting the training e ort of the initial network into several networks and multi-classi ers namely a surface network and bottom network are used to solve the problem and to enhance the performance. All methods are evaluated in a MATLAB framework and confusion matrices and average facial expression recognition accuracy are used as the performance metrics.