  Institutions can provide incentives to increase cooperation behaviour in a
population where this behaviour is infrequent. This process is costly, and it
is thus important to optimize the overall spending. This problem can be
mathematically formulated as a multi-objective optimization problem where one
wishes to minimize the cost of providing incentives while ensuring a desired
level of cooperation within the population. In this paper, we provide a
rigorous analysis for this problem. We study cooperation dilemmas in both the
pairwise (the Donation game) and multi-player (the Public Goods game) settings.
We prove the regularity of the (total incentive) cost function, characterize
its asymptotic limits (infinite population, weak selection and large selection)
and show exactly when reward or punishment is more efficient. We prove that the
cost function exhibits a phase transition phenomena when the intensity of
selection varies. We calculate the critical threshold in regards to the phase
transition and study the optimization problem when the intensity of selection
is under and above the critical value. It allows us to provide an exact
calculation for the optimal cost of incentive, for a given intensity of
selection. Finally, we provide numerical simulations to demonstrate the
analytical results. Overall, our analysis provides for the first time a
selection-dependent calculation of the optimal cost of institutional incentives
(for both reward and punishment) that guarantees a minimum amount of
cooperation. It is of crucial importance for real-world applications of
institutional incentives since intensity of selection is specific to a given
population and the underlying game payoff structure.
