Optimal Spot-Checking for Improving Evaluation Accuracy of Peer Grading Systems

Authors

  • Wanyuan Wang Southeast University
  • Bo An Nanyang Technological Univeristy
  • Yichuan Jiang Southeast University

Keywords:

Aritificial Intelligence, Multiagent Systems, Game-Theory, Optimization

Abstract

Peer grading, allowing students/peers to evaluate others' assignments, offers a promising solution for scaling evaluation and learning to large-scale educational systems. A key challenge in peer grading is motivating peers to grade diligently. While existing spot-checking (SC) mechanisms can prevent peer collusion where peers coordinate to report the uninformative grade, they unrealistically assume that peers have the same grading reliability and cost. This paper studies the general Optimal Spot-Checking (OptSC) problem of determining the probability each assignment needs to be checked to maximize assignments' evaluation accuracy aggregated from peers, and takes into consideration 1) peers' heterogeneous characteristics, and 2) peers' strategic grading behaviors to maximize their own utility. We prove that the bilevel OptSC is NP-hard to solve. By exploiting peers' grading behaviors, we first formulate a single level relaxation to approximate OptSC. By further exploiting structural properties of the relaxed problem, we propose an efficient algorithm to that relaxation, which also gives a good approximation of the original OptSC. Extensive experiments on both synthetic and real datasets show significant advantages of the proposed algorithm over existing approaches.

Downloads

Published

2018-04-25

How to Cite

Wang, W., An, B., & Jiang, Y. (2018). Optimal Spot-Checking for Improving Evaluation Accuracy of Peer Grading Systems. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/11336

Issue

Section

Computational Sustainability and Artificial Intelligence