Human–computer interaction (HCI) has received growing interest in both academic research and the design of information technological applications. Automated facial expression estimation of image is a difficult, yet crucial, problem in the design of HCI system. Although artificial neural network has achieved many remarkable results, few smart wearable devices can benefit from it. Most of these devices are constrained by limited computing and storage capacity. An effective solution is to allow servers to handle multiple tasks simultaneously. Toward this goal, we have been building an Efficient multitask scheme for facial expression estimation (EM-FEE). A multitask neural network is designed to enable the HCI system to accomplish different related tasks at the same time, that is, locating the user’s facial landmarks and estimating facial expressions. Experimental results demonstrate that our proposed scheme outperforms state-of-the-art. Finally, we review the remaining challenges and corresponding opportunities as well as future directions of the design of facial expression estimation systems for smart wearable devices.@en