Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

Authors

  • Ivan Stelmakh Carnegie Mellon University
  • Nihar B. Shah Carnegie Mellon University
  • Aarti Singh Carnegie Mellon University

Keywords:

AI for Conference Organization and Delivery (AICOD), Adversarial Agents

Abstract

We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-assessment task is competitive (e.g., when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering. In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour. We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings. For this, we design and conduct an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest. We use this data to run a series of real and semi-synthetic evaluations that reveal a strong detection power of our test.

Downloads

Published

2021-05-18

How to Cite

Stelmakh, I., Shah, N. B., & Singh, A. (2021). Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment. Proceedings of the AAAI Conference on Artificial Intelligence, 35(6), 4794-4802. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16611

Issue

Section

AAAI Technical Track Focus Area on AI for Conference Organization and Delivery