Multistakeholder recommender systems are those that account for the impacts and preferences of multiple groups of individuals, not just the end users receiving recommendations. Due to their complexity, these systems cannot be evaluated strictly by the overall utility of a single stakeholder, as is often the case of more mainstream recommender system applications. In this article, we focus our discussion on the challenges of multistakeholder evaluation of recommender systems. We bring attention to the different aspects involved—from the range of stakeholders involved (including but not limited to providers and consumers) to the values and specific goals of each relevant stakeholder. We discuss how to move from theoretical principles to practical implementation, providing specific use case examples. Finally, we outline open research directions for the RecSys community to explore. We aim to provide guidance to researchers and practitioners about incorporating these complex and domain-dependent issues of evaluation in the course of designing, developing, and researching applications with multistakeholder aspects.

During the past decade, recommender systems have rapidly become an indispensable element of websites, apps, and other platforms that seek to provide personalized interactions to their users. As recommendation technologies are applied to an ever-growing array of non-standard problems and scenarios, researchers and practitioners are also increasingly faced with challenges of dealing with greater variety and complexity in the inputs to those recommender systems. For example, there has been more reliance on fine-grained user signals as inputs rather than simple ratings or likes. Applications require more complex domain-specific constraints on inputs to the recommender systems. Likewise, the outputs of recommender systems are moving towards more complex composite items, such as package or sequence recommendations. This increasing complexity requires smarter recommender algorithms that can deal with this diversity in inputs and outputs. For the past four years, the ComplexRec workshop series has offered an interactive venue for discussing approaches to recommendation in complex scenarios that have no simple one-size-fits-all solution.