ST-Sem

ST-Sem: A Multimodal Method for Points-of-Interest Classification Using Street-Level Imagery

Sharifi Noorian, S. (TU Delft Web Information Systems)
Psyllidis, A. (TU Delft Web Information Systems)
Bozzon, A. (TU Delft Web Information Systems)

Bakaev, Maxim (editor)
Ko, In-Young (editor)
Frasincar, Flavius (editor)

2019-04-26

Street-level imagery contains a variety of visual information about the facades of Points of Interest (POIs). In addition to general mor- phological features, signs on the facades of, primarily, business-related POIs could be a valuable source of information about the type and iden- tity of a POI. Recent advancements in computer vision could leverage visual information from street-level imagery, and contribute to the classification of POIs. However, there is currently a gap in existing literature regarding the use of visual labels contained in street-level imagery, where their value as indicators of POI categories is assessed. This paper presents Scene-Text Semantics (ST-Sem), a novel method that leverages visual la- bels (e.g., texts, logos) from street-level imagery as complementary in- formation for the categorization of business-related POIs. Contrary to existing methods that fuse visual and textual information at a feature- level, we propose a late fusion approach that combines visual and textual cues after resolving issues of incorrect digitization and semantic ambiguity of the retrieved textual components. Experiments on two existing and a newly-created datasets show that ST-Sem can outperform visual-only approaches by 80% and related multimodal approaches by 4%.

Convolutional neural networks
Points of Interest
Semantic similarity
Street-level imagery
Word embeddings

http://resolver.tudelft.nl/uuid:a651873c-5aed-4824-8281-308d98f3f9d1

https://doi.org/10.1007/978-3-030-19274-7_3

Springer, Cham

978-3-030-19273-0

Web Engineering - 19th International Conference, ICWE 2019, Proceedings, 11496

19th International Conference on Web Engineering, ICWE 2019, 2019-06-11 → 2019-06-14, Daejeon Convention Center (DCC), Daejeon, Korea, Republic of

Lecture Notes in Computer Science, 0302-9743, 11496

Institutional Repository

conference paper

© 2019 S. Sharifi Noorian, A. Psyllidis, A. Bozzon