text: We consider crowdsourcing problems where the users are asked to provide evaluations for items; the user evaluations are then used directly, or aggregated into a consensus value. Lacking an incentive scheme, users have no motive in making effort in completing the evaluations, providing inaccurate answers instead. We propose incentive schemes that are truthful and cheap: truthful as the optimal user behavior consists in providing accurate evaluations, and cheap because the truthfulness is achieved with little overhead cost. We consider both discrete evaluation tasks, where an evaluation can be done either correctly, or incorrectly, with no degrees of approximation in between, and quantitative evaluation tasks, where evaluations are real numbers, and the error is measured as distance from the correct value. For both types of tasks, we propose hierarchical incentive schemes that can be effected with a small amount of additional evaluations, and that scale to arbitrarily large crowd sizes: they have the property that the strength of the incentive does not weaken with increasing hierarchy depth. Interestingly, we show that for these schemes to work, the only requisite is that workers know their place in the hierarchy in advance.
  @cite_0: Individuals with finite private information independently choose acts and messages. Their utilities may depend on all acts and information, including the center's. Incentive payments are separable and fully transferable. Implementable incentives making specified behavior a Bayesian equilibrium are derived whenever the center's information depends stochastically, however slightly, on all relevant private information, and also whenever individuals' relative valuations of acts, however divergent, are not too dissimilarly affected by different states of nature. Feasibility is resolved whenever the desired strategies reveal the agents' beliefs about the center's information. Key concepts of agent similarity are developed for nonresponsive and budget-balancing cases. Copyright 1990 by The Econometric Society.
  @cite_13: Abstract Since a meteorologist's predictions are subjective, a framework for the evaluation of meteorological probability assessors must be consistent with the theory of subjective probability. Such a framework is described in this paper. First, two standards of “goodness,” one based upon normative considerations and one based upon substantive considerations, are proposed. Specific properties which a meteorologist's assessments should possess are identified for each standard. Then, several measures of “goodness,” or scoring rules, which indicate the extent to which such assessments possess certain properties, are described. Finally, several important uses of these scoring rules are considered.
  @cite_20: When a decision maker (DM) contracts with an expert to provide information, the nature of the contract can, create incentives for the expert, and it is up to the DM to ensure that the contract provides incentives that align the expert’s and DM’s interests. In this paper, scoring rules (and related functions) are viewed as such contracts and are reinterpreted in terms of agency theory and the theory of revelation games from economics. Although scoring rules have typically been discussed in the literature as devices for eliciting and evaluating subjective probabilities, this study relies on the fact that strictly proper scoring rules reward greater expertise as well as honest revelation. We describe conditions under which a DM can use a strictly proper scoring rule as a contract to give an expert an incentive to gather an amount of information that is optimal from the DM’s perspective. The conditions we consider focus on the expert’s cost structure, and we find that the DM must have substantial knowledge of that cost structure in order to design a specific contract that provides the correct incentives. The model and analysis suggest arguments for hiring and maintaining experts in-house rather than using outside consultants.
summary: Prediction markets are models with a goal of obtaining predictions about events of interest from experts. After experts provide predictions, a system assigns a reward based on a scoring rule to every expert. Proper scoring rules ensure that the highest reward is achieved by reporting the true probability distribution @cite_13 @cite_0 @cite_20 . An assumption of the scoring rules is that the future outcome must be observable. This assumption prevents crowdsourcing systems to scale to large crowds as obtaining the correct answer for each event or task is prohibitively expensive.