This paper focuses on fine-grained object classification using recognized scene text in natural images. While the state-of-the-art relies on visual cues only, this paper is the first work which proposes to combine textual and visual cues. Another novelty is the textual cue extraction. Unlike the state-of-the-art text detection methods, we focus more on the background instead of text regions. Once text regions are detected, they are further processed by two methods to perform text recognition, i.e., ABBYY commercial OCR engine and a state-of-the-art character recognition algorithm. Then, to perform textual cue encoding, bi- and trigrams are formed between the recognized characters by considering the proposed spatial pairwise constraints. Finally, extracted visual and textual cues are combined for fine-grained classification. The proposed method is validated on four publicly available data sets: ICDAR03, ICDAR13, Con-Text, and Flickr-logo. We improve the state-of-the-art end-to-end character recognition by a large margin of 15% on ICDAR03. We show that textual cues are useful in addition to visual cues for fine-grained classification. We show that textual cues are also useful for logo retrieval. Adding textual cues outperforms visual- and textual-only in fine-grained classification (70.7% to 60.3%) and logo retrieval (57.4% to 54.8%).

This work considers the task of object proposal scoring by integrating the consistency between state- of-the-art object proposal algorithms. It represents a novel way of thinking about proposals, as it starts with the assumption that consistent proposals are most likely centered on objects in the image. We pose the box-consistency problem as a large-scale regression task. The approach starts from existing popular object proposal algorithms and assigns scores to these proposals based on the consistency within and be- tween algorithms. Rather than generating new proposals, we focus on the consistency of state-of-the-art ones and score them on the assumption that mutually agreeing proposals usually indicate the location of objects. This work performs large-scale regression by starting from the strong Gaussian Process model, renowned for its power as a regressor. We extend the model in a natural manner to make effective use of the large number of training samples. We achieve this through metric learning for reshaping the kernel space, while maintaining the kernel-matrix size fixed. We validated the new Gaussian Process models on a standard regression dataset —Airfoil Self-Noise —to prove the generality of the method. Further- more, we test the suitability of the proposed approach for the undertaken box scoring task on Pascal- VOC2007. We conclude that box scoring is possible by employing overlap statistics in a new Gaussian Process model, fine tuned to handle large amounts of data.