Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)
Ryan Clancy (University of Waterloo)
Nicola Ferro (Università degli Studi di Padova)
Claudia Hauff (TU Delft - Web Information Systems)
Jimmy Lin (University of Waterloo)
Tetsuya Sakai (Waseda University)
Ze Zhong Wu (University of Waterloo)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.
help_outline