Peer grading the peer reviews: A dual-role approach for lightening the scholarly paper review process

Arous, Ines; Yang, J.; Khayati, Mourad; Cudre-Mauroux, Philippe

doi:10.1145/3442381.3450088

Peer grading the peer reviews

Title

Peer grading the peer reviews: A dual-role approach for lightening the scholarly paper review process

Author

Arous, Ines (University of Fribourg)
Yang, J. (TU Delft Web Information Systems)
Khayati, Mourad (University of Fribourg)
Cudre-Mauroux, Philippe (University of Fribourg)

Date

2021

Abstract

Scientific peer review is pivotal to maintain quality standards for academic publication. The effectiveness of the reviewing process is currently being challenged by the rapid increase of paper submissions in various conferences. Those venues need to recruit a large number of reviewers of different levels of expertise and background. The submitted reviews often do not meet the conformity standards of the conferences. Such a situation poses an ever-bigger burden on the meta-reviewers when trying to reach a final decision. In this work, we propose a human-AI approach that estimates the conformity of reviews to the conference standards. Specifically, we ask peers to grade each other's reviews anonymously with respect to important criteria of review conformity such as sufficient justification and objectivity. We introduce a Bayesian framework that learns the conformity of reviews from both the peer grading process, historical reviews and decisions of a conference, while taking into account grading reliability. Our approach helps meta-reviewers easily identify reviews that require clarification and detect submissions requiring discussions while not inducing additional overhead from reviewers. Through a large-scale crowdsourced study where crowd workers are recruited as graders, we show that the proposed approach outperforms machine learning or review grades alone and that it can be easily integrated into existing peer review systems.

Subject

Crowdsourcing
Human-AI collaboration
Peer grading
Peer review

To reference this document use:

http://resolver.tudelft.nl/uuid:efd06fd5-b732-4de2-8bc5-af985d5c5236

DOI

https://doi.org/10.1145/3442381.3450088

Publisher

Association for Computing Machinery (ACM)

ISBN

978-1-4503-8312-7

Source

The Web Conference 2021 - Proceedings of the World Wide Web Conference, WWW 2021