In real-world scenarios, recommenders face non-functional requirementsof technical nature and must handle dynamic data in the formof sequential streams. Evaluation of recommender systems musttake these issues into account in order to be maximally informative.In this paper, we present Idomaar—a framework that enables theefficient multi-dimensional benchmarking of recommender algorithms.Idomaar goes beyond current academic research practicesby creating a realistic evaluation environment and computing botheffectiveness and technical metrics for stream-based as well as setbasedevaluation. A scenario focussing on “research to prototypingto productization” cycle at a company illustrates Idomaar’s potential.We show that Idomaar simplifies testing with varying configurationsand supports flexible integration of different data.@en

Successful news recommendation requires facing the challenges of dynamic item sets, contextual item relevance, and of fulfilling non-functional requirements, such as response time. The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to tackle news recommendation and to optimize and evaluate their recommender algorithms both online and offline. In this paper, we summarize the objectives and challenges of NewsREEL 2016. We cover two contrasting perspectives on the challenge: that of the operator (the business providing recommendations) and that of the challenge participant (the researchers developing recommender algorithms). In the intersection of these perspectives, new insights can be gained on how to effectively evaluate real-time stream recommendation algorithms.@en

Running in its third year at CLEF, NewsREEL challenged participantsto develop news recommendation algorithms and have them benchmarked inan online (Task 1) and offline setting (Task 2), respectively. This paper providesan overview of the NewsREEL scenario, outlines this year’s campaign, presentsresults of both tasks, and discusses the approaches of participating teams. Moreover,it overviews ideas on living lab evaluation that have been presented as partof a “New Ideas” track at the conference in Portugal. Presented results illustratepotentials for multi-dimensional evaluation of recommendation algorithms ina living lab and simulation based evaluation setting.@en