text: Peer grading systems work well only if users have incentives to grade truthfully. An example of non-truthful grading, that we observed in classrooms, consists in students assigning the maximum grade to all submissions. With a naive grading scheme, such as averaging the assigned grades, all students would receive the maximum grade. In this paper, we develop three grading schemes that provide incentives for truthful peer grading. In the first scheme, the instructor grades a fraction p of the submissions, and penalizes students whose grade deviates from the instructor grade. We provide lower bounds on p to ensure truthfulness, and conclude that these schemes work only for moderate class sizes, up to a few hundred students. To overcome this limitation, we propose a hierarchical extension of this supervised scheme, and we show that it can handle classes of any size with bounded (and little) instructor work, and is therefore applicable to Massive Open Online Courses (MOOCs). Finally, we propose unsupervised incentive schemes, in which the student incentive is based on statistical properties of the grade distribution, without any grading required by the instructor. We show that the proposed unsupervised schemes provide incentives to truthful grading, at the price of being possibly unfair to individual students.
  @cite_9: The success of a human computation system depends critically on the humans in the system actually behaving, or acting, as necessary for the system to function effectively. Since users have their own costs and benefits from participation, they will undertake desirable actions only if properly incentivized to do so: Indeed, while there are a vast number of human computation systems on the Web, the extent of participation and quality of contribution varies widely across systems. How can a game-theoretic approach help understand why, and provide guidance on designing systems that incentivize high participation and effort from contributors?
summary: Providing incentives to human agents to return thuthful responses is a central challenge of crowdsourcing algorithms and applications @cite_9 .