Multistakeholder recommender systems are those that account for the impacts and preferences of multiple groups of individuals, not just the end users receiving recommendations. Due to their complexity, these systems cannot be evaluated strictly by the overall utility of a single stakeholder, as is often the case of more mainstream recommender system applications. In this article, we focus our discussion on the challenges of multistakeholder evaluation of recommender systems. We bring attention to the different aspects involved—from the range of stakeholders involved (including but not limited to providers and consumers) to the values and specific goals of each relevant stakeholder. We discuss how to move from theoretical principles to practical implementation, providing specific use case examples. Finally, we outline open research directions for the RecSys community to explore. We aim to provide guidance to researchers and practitioners about incorporating these complex and domain-dependent issues of evaluation in the course of designing, developing, and researching applications with multistakeholder aspects.

The goal of the present study was to investigate how satisfied individuals are with the final outcome of a group decision-making process on a joint travel destination. Using an experimental paradigm (N total = 200, N groups = 55) it was obvious to hypothesize that individuals would especially be satisfied with the final group decision when it matched their own initial travel preference and that they would be dissatisfied in case it mismatched their initial preference. However, in addition the influence of personality and group dynamics differences (Thomas-Kilmann Conflict Mode Instrument, Five Factor Model) as well as travel types of the individual decision maker on the satisfaction level with the group decision outcome as the dependent variable were further researched. The paper concludes with implications for e-tourism, especially with regards to the development of interactive tools for group travel.

Most research on group recommender systems relies on the assumption that individuals have conflicting preferences; in order to generate group recommendations the system should identify a fair way of aggregating these preferences. Both empirical studies and theoretical frameworks have tried to identify the most effective preference aggregation techniques without coming to definite conclusions. In this paper, we propose to approach group recommendation from the group dynamics perspective and analyze the group decision making process for a particular task (in the travel domain). We observe several individual and group properties and correlate them to choice satisfaction. Supported by these initial results we therefore advocate for the development of new group recommendation techniques that consider group dynamics and support the full group decision making process.