Print Email Facebook Twitter From Deterministic to Generative Title From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning Author Song, Jingkuan (University of Electronic Science and Technology of China) Guo, Yuyu (University of Electronic Science and Technology of China) Gao, Lianli (University of Electronic Science and Technology of China) Li, Xuelong (Chinese Academy of Sciences) Hanjalic, A. (TU Delft Intelligent Systems) Shen, Heng Tao (University of Electronic Science and Technology of China) Department Intelligent Systems Date 2018 Abstract Video captioning, in essential, is a complex natural process, which is affected by various uncertainties stemming from video content, subjective judgment, and so on. In this paper, we build on the recent progress in using encoder-decoder framework for video captioning and address what we find to be a critical deficiency of the existing methods that most of the decoders propagate deterministic hidden states. Such complex uncertainty cannot be modeled efficiently by the deterministic models. In this paper, we propose a generative approach, referred to as multimodal stochastic recurrent neural networks (MS-RNNs), which models the uncertainty observed in the data using latent stochastic variables. Therefore, MS-RNN can improve the performance of video captioning and generate multiple sentences to describe a video considering different random factors. Specifically, a multimodal long short-term memory (LSTM) is first proposed to interact with both visual and textual features to capture a high-level representation. Then, a backward stochastic LSTM is proposed to support uncertainty propagation by introducing latent variables. Experimental results on the challenging data sets, microsoft video description and microsoft research video-to-text, show that our proposed MS-RNN approach outperforms the state-of-the-art video captioning benchmarks. Subject Recurrent neural network (RNN)uncertaintyvideo captioning. To reference this document use: http://resolver.tudelft.nl/uuid:953b6795-560d-470e-809e-0dda843ecc68 DOI https://doi.org/10.1109/TNNLS.2018.2851077 ISSN 2162-237X Source IEEE Transactions on Neural Networks and Learning Systems, 1-12 Bibliographical note Accepted Author Manuscript Part of collection Institutional Repository Document type journal article Rights © 2018 Jingkuan Song, Yuyu Guo, Lianli Gao, Xuelong Li, A. Hanjalic, Heng Tao Shen Files PDF 46671202_08438512.pdf 3.28 MB Close viewer /islandora/object/uuid:953b6795-560d-470e-809e-0dda843ecc68/datastream/OBJ/view