Idomaar

A Framework for Multi-dimensional Benchmarking of Recommender Algorithms

Conference paper (2016)

Authors

Mario Scriminaci ContentWise R&D
A Lommatzsch Technical University of Berlin
B Kille Technical University of Berlin
Frank Hopfgartner University of Glasgow
Davide Malagoli ContentWise R&D
András Serény Gravity Research
Till Plumbaum Technical University of Berlin
More Info
expand_more

Published Date

2016-9

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

In real-world scenarios, recommenders face non-functional requirementsof technical nature and must handle dynamic data in the formof sequential streams. Evaluation of recommender systems musttake these issues into account in order to be maximally informative.In this paper, we present Idomaar—a framework that enables theefficient multi-dimensional benchmarking of recommender algorithms.Idomaar goes beyond current academic research practicesby creating a realistic evaluation environment and computing botheffectiveness and technical metrics for stream-based as well as setbasedevaluation. A scenario focussing on “research to prototypingto productization” cycle at a company illustrates Idomaar’s potential.We show that Idomaar simplifies testing with varying configurationsand supports flexible integration of different data.