Title
Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO
Author
Chen, Yangkun (Tsinghua University; Parametrix.ai)
Yu, Chenghui (Tsinghua University; Parametrix.ai)
Zhu, Hengman (Parametrix.ai)
Liu, Shuai (Bilibili)
Zhang, Yibing (Chengdu Goldwin Electronics Technology)
Suarez, Joseph (Massachusetts Institute of Technology)
Zhao, Liang (International Digital Economy Academy)
He, J. (TU Delft Interactive Intelligence)
Chen, Jiaxin (Parametrix.ai)
Date
2023
Abstract
We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.
Subject
Benchmark
Competition
Multi-agent Reinforcement Learning
To reference this document use:
http://resolver.tudelft.nl/uuid:311c668f-8cb7-4201-8105-c19ab87aa888
Embargo date
2023-11-27
ISSN
1548-8403
Source
Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2023-May, 2490-2492
Event
22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023, 2023-05-29 → 2023-06-02, London, United Kingdom
Bibliographical note
Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Part of collection
Institutional Repository
Document type
journal article
Rights
© 2023 Yangkun Chen, Chenghui Yu, Hengman Zhu, Shuai Liu, Yibing Zhang, Joseph Suarez, Liang Zhao, J. He, Jiaxin Chen, More Authors