Synthesizing Spoken Descriptions of Images

Synthesizing Spoken Descriptions of Images

Wang, X. (TU Delft Multimedia Computing; Xi’an Jiaotong University)
van der Hout, Justin (Student TU Delft)
Zhu, Jihua (Xi’an Jiaotong University)
Hasegawa-Johnson, Mark (University of Illinois at Urbana Champaign)
Scharenborg, O.E. (TU Delft Multimedia Computing)

2021

Image captioning technology has great potential in many scenarios. However, current text-based image captioning methods cannot be applied to approximately half of the world's languages due to these languages’ lack of a written form. To solve this problem, recently the image-to-speech task was proposed, which generates spoken descriptions of images bypassing any text via an intermediate representation consisting of phonemes (image-to-phoneme). Here, we present a comprehensive study on the image-to-speech task in which, 1) several representative image-to-text generation methods are implemented for the image-to-phoneme task, 2) objective metrics are sought to evaluate the image-to-phoneme task, and 3) an end-to-end image-to-speech model that is able to synthesize spoken descriptions of images bypassing both text and phonemes is proposed. Extensive experiments are conducted on the public benchmark database Flickr8k. Results of our experiments demonstrate that 1) State-of-the-art image-to-text models can perform well on the image-to-phoneme task, and 2) several evaluation metrics, including BLEU3, BLEU4, BLEU5, and ROUGE-L can be used to evaluate image-to-phoneme performance. Finally, 3) end-to-end image-to-speech bypassing text and phonemes is feasible.

Speech processing
Image-to-speech generation
multimodal modelling
speech synthesis
cross-modal captioning

http://resolver.tudelft.nl/uuid:914070cb-d4fe-4bad-8064-bc83154b895b

https://doi.org/10.1109/TASLP.2021.3120644

2329-9304

IEEE - ACM Transactions on Audio, Speech, and Language Processing, 29, 3242-3254

Accepted author manuscript

Institutional Repository

journal article

© 2021 X. Wang, Justin van der Hout, Jihua Zhu, Mark Hasegawa-Johnson, O.E. Scharenborg