rewards more heavily, conversely by reducing it we weight them less so. In other words, this parameter controls how short term greedy, or long term farsighted, the agent should be.

To work out the expected future value for a given agent and environment interacting, we take the sum of these discounted rewards into the infinite future and work out its expected value,

$$V_{\mu}^{\pi}(\gamma):=\,\frac{1}{\Gamma}\mathbf{E}\left(\sum_{i=1}^{\infty}\gamma^{i}r_{i}\right).$$
$$\left(1\right)$$

In the above, riis the reward in cycle i of a given history, γ is the discount rate, γ iis the discount applied to the i th reward into the future, the normalising constant is Γ := P∞
i=1 γ i, and the expected value is taken over all possible interaction sequences between the agent π and the environment µ.

Under geometric discounting an agent with γ = 0.95 will not plan further than about 20 cycles ahead. Thus we say that the agent has a constant effective horizon of 1 1−γ
.

Since we are interested in universal intelligence, a limited farsightedness is not acceptable because for every horizon there is a task that needs a larger horizon to be solved. For instance, while a horizon of 5 is sufficient for tic-tac-toe, it is insufficient for chess. Clearly, geometric discounting has not solved the problem of how to weight near term rewards versus long term rewards, it has simply expressed this weighting as a parameter. What we require is a single definition of machine intelligence, not a range of definitions that vary according to a free parameter.

A more promising candidate for universal discounting is the near-harmonic, or quadratic discount, where we replace γ iin Equation 1 by 1/i2 and modifying Γ accordingly. This has some interesting properties, in particular the agent needs to look forward into the future in a way that is proportional to its current age. This is appealing since it seems that humans of age k years usually do not plan their lives for more than, perhaps, the next k years. More importantly, it allows us to avoid the problem of having to choose a global time scale or effective horizon [Hut05].

Although harmonic discounting has a number of attractive properties [Hut06a], an even simpler and more general solution is possible. If we look at the value function in Equation 1, we see that discounting plays two roles. Firstly, it normalises rewards received so that their sum is always finite. Secondly, it weights the reward at different points in the future which in effect defines a temporal preference. A direct way to solve both of these problems, without needing an external parameter, is to simply require that the total reward returned by the environment can never exceed 1. For such a reward summable environment µ, it follows that the expected value of the sum of rewards is also finite and thus discounting is no longer required,

$$V_{\mu}^{\pi}:=\,\mathbf{E}\left(\sum_{i=1}^{\infty}r_{i}\right)\leq1.$$
$$\left(2\right)$$
≤ 1. (2)
One way of viewing this is that the rewards returned by the environment now have the temporal preference already factored in. The cost is that this is an additional condition that we place on the space of environments. Previously we required that each reward signal was in a subset of [0, 1]∩Q, now we have the additional constraint that the reward sum is always bounded (see Subsection 5.2 for further discussion about why this constraint is reasonable).