The strange case of reproducibility versus representativeness in contextual suggestion test collections

The most common approach to measuring the effectiveness of Information Retrieval systems is by using test collections. The Contextual Suggestion (CS) TREC track provides an evaluation framework for systems that recommend items to users given their geographical context. The specific nature of this track allows the participating teams to identify...

Increasing cheat robustness of crowdsourcing tasks

Crowdsourcing successfully strives to become a widely used means of collecting large-scale scientific corpora. Many research fields, including Information Retrieval, rely on this novel way of data acquisition. However, it seems to be undermined by a significant share of workers that are primarily interested in producing quick generic answers...

Reliability and effectiveness of clickthrough data for automatic image annotation

Automatic image annotation using supervised learning is performed by concept classifiers trained on labelled example images. This work proposes the use of clickthrough data collected from search logs as a source for the automatic generation of concept training data, thus avoiding the expensive manual annotation effort. We investigate and...

Eye Movements Disclose Decisions in Set

During the past several decades, much research has been done in the field of visual attention and cognitive processing, i.e., how does the human brain process visual stimuli. We show that the card game SET is a good candidate for investigating the relation between top-down versus bottom-up visual information processing. Further, we show that...