Optical flow-based facial feature tracking to recognize AUs modeled by Bayesian networks

Abstract

This thesis describes a Bayesian Network (BN) model for recognizing the “Action Units (AUs)” of a facial expression using video sequence images as input. Features were extracted by using an optimal estimation optical flow method coupled with a physical (muscle) model describing the facial structure. The muscle action patterns are used for analysis, recognition, and synthesis of facial expressions. In the thesis the main approaches to facial expression recognition of dynamic images are designed considering three main parts: 1) Region of Interest Selection, 2) Feature Extraction, and 3) Image Classification. Bayesian Networks are a powerful and flexible methodology for representing and computing with probabilistic models of a stochastic process. In the past decade, there has been increasing interest in applying them to practical problems, and this thesis shows that they can be used effectively in the field of automatic AU’s recognition. In past decade optical flows have been used to either model muscle activities or estimate the displacements of feature points but in this thesis we defined nine regions of interest (ROI) which contains the most complex motion by using entropy maximum algorithm. Furthermore, the results were statistically analyzed by compass diagrams to find out the major ranges of directions and velocities of vector flows in each ROI. We found that for the six basic emotions, the ROI are different, so we did not consider all of nine regions for every emotion because of the complexity of our model. Furthermore, we present a methodology for obtaining the BN structure, learning the parameters and inference, including issues such as the discretization of continuous variables. Finally, we apply the BN model to recognize single Action Units (AUs) and some important AU combinations. The average classification rate for the single AUs is between 80% and 90% and for the AU combinations is above 90%