text: Peer grading systems work well only if users have incentives to grade truthfully. An example of non-truthful grading, that we observed in classrooms, consists in students assigning the maximum grade to all submissions. With a naive grading scheme, such as averaging the assigned grades, all students would receive the maximum grade. In this paper, we develop three grading schemes that provide incentives for truthful peer grading. In the first scheme, the instructor grades a fraction p of the submissions, and penalizes students whose grade deviates from the instructor grade. We provide lower bounds on p to ensure truthfulness, and conclude that these schemes work only for moderate class sizes, up to a few hundred students. To overcome this limitation, we propose a hierarchical extension of this supervised scheme, and we show that it can handle classes of any size with bounded (and little) instructor work, and is therefore applicable to Massive Open Online Courses (MOOCs). Finally, we propose unsupervised incentive schemes, in which the student incentive is based on statistical properties of the grade distribution, without any grading required by the instructor. We show that the proposed unsupervised schemes provide incentives to truthful grading, at the price of being possibly unfair to individual students.
  @cite_21: Individuals with finite private information independently choose acts and messages. Their utilities may depend on all acts and information, including the center's. Incentive payments are separable and fully transferable. Implementable incentives making specified behavior a Bayesian equilibrium are derived whenever the center's information depends stochastically, however slightly, on all relevant private information, and also whenever individuals' relative valuations of acts, however divergent, are not too dissimilarly affected by different states of nature. Feasibility is resolved whenever the desired strategies reveal the agents' beliefs about the center's information. Key concepts of agent similarity are developed for nonresponsive and budget-balancing cases. Copyright 1990 by The Econometric Society.
  @cite_12: A challenge with the programmatic access of human talent via crowdsourcing platforms is the specification of incentives and the checking of the quality of contributions. Methodologies for checking quality include providing a payment if the work is approved by the task owner and hiring additional workers to evaluate contributors' work. Both of these approaches place a burden on people and on the organizations commissioning tasks, and may be susceptible to manipulation by workers and task owners. Moreover, neither a task owner nor the task market may know the task well enough to be able to evaluate worker reports. Methodologies for incentivizing workers without external quality checking include rewards based on agreement with a peer worker or with the final output of the system. These approaches are vulnerable to strategic manipulations by workers. Recent experiments on Mechanical Turk have demonstrated the negative influence of manipulations by workers and task owners on crowdsourcing systems [3]. We address this central challenge by introducing incentive mechanisms that promote truthful reporting in crowdsourcing and discourage manipulation by workers and task owners without introducing additional overhead.
summary: The peer-prediction method @cite_21 uses proper scoring rules to reward experts depending on how good their input for predicting other experts' reports. Similarly, the model described in @cite_12 evaluates experts depending on how good their reports are in predicting the consensus of other workers.