Benchmarking Robustness and Generalization in Multi-Agent Systems
A Case Study on Neural MMO
Yangkun Chen (Parametrix.ai, Tsinghua University)
Chenghui Yu (Parametrix.ai, Tsinghua University)
Hengman Zhu (Parametrix.ai)
Shuai Liu (Bilibili)
Yibing Zhang (Chengdu Goldwin Electronics Technology)
Joseph Suarez (Massachusetts Institute of Technology)
Liang Zhao (International Digital Economy Academy)
Jinke He (TU Delft - Interactive Intelligence)
Jiaxin Chen (Parametrix.ai)
undefined More Authors (External organisation)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.