The explosion of video data in surveillance calls for large amount of annotated datasets that could be used for information retrieval, learning-based network training and algorithms evaluation phase. A number of annotated video datasets have been shared to public, however, these open annotated datasets lack spatio-temporal information which can contribute to motion analysis researches. Moreover, manual annotation is tedious in lengthy frame sequences where the numbers are overwhelming. In this work we propose a visualization solution to facilitate video data retrieval, mining and annotation. It works as an integrated visual analytics system which supports collecting moving object samples and studying motion patterns in video, with the facilitation of video data visualization, using various spatial and/or temporal features, filtering parameters and similarity measuring models. The annotation output can be applied to multiple video analysis researches. In this paper, we present the system and propose a workflow for costeffective video annotation of spatio-temporal data as well as facilitating comprehension of video data.