Print Email Facebook Twitter DaisyRec 2.0 Title DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation Author Sun, Zhu (Institute of High Performance Computing) Fang, Hui Yang, J. (TU Delft Web Information Systems) Qu, Xinghua (Bytedance AI Lab) Liu, Hongyang (Yanshan University) Yu, Di (Singapore Management University) Ong, Yew Soon (Nanyang Technological University) Zhang, Jie (Nanyang Technological University) Date 2023 Abstract Recently, one critical issue looms large in the field of recommender systems - there are no effective benchmarks for rigorous evaluation - which consequently leads to unreproducible evaluation and unfair comparison. We, therefore, conduct studies from the perspectives of practical theory and experiments, aiming at benchmarking recommendation for rigorous evaluation. Regarding the theoretical study, a series of hyper-factors affecting recommendation performance throughout the whole evaluation chain are systematically summarized and analyzed via an exhaustive review on 141 papers published at eight top-tier conferences within 2017-2020. We then classify them into model-independent and model-dependent hyper-factors, and different modes of rigorous evaluation are defined and discussed in-depth accordingly. For the experimental study, we release DaisyRec 2.0 library by integrating these hyper-factors to perform rigorous evaluation, whereby a holistic empirical study is conducted to unveil the impacts of different hyper-factors on recommendation performance. Supported by the theoretical and experimental studies, we finally create benchmarks for rigorous evaluation by proposing standardized procedures and providing performance of ten state-of-the-arts across six evaluation metrics on six datasets as a reference for later study. Overall, our work sheds light on the issues in recommendation evaluation, provides potential solutions for rigorous evaluation, and lays foundation for further investigation. Subject Benchmarksfair comparisonrecommender systemsreproducible evaluationstandardized procedures To reference this document use: http://resolver.tudelft.nl/uuid:1014a9d2-3604-46ac-a92e-cb6e485e7d72 DOI https://doi.org/10.1109/TPAMI.2022.3231891 Embargo date 2023-06-26 ISSN 0162-8828 Source IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (7), 8206-8226 Bibliographical note Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. Part of collection Institutional Repository Document type journal article Rights © 2023 Zhu Sun, Hui Fang, J. Yang, Xinghua Qu, Hongyang Liu, Di Yu, Yew Soon Ong, Jie Zhang Files PDF DaisyRec_2.0_Benchmarking ... uation.pdf 3.74 MB Close viewer /islandora/object/uuid:1014a9d2-3604-46ac-a92e-cb6e485e7d72/datastream/OBJ/view