
## 1. Abstract

We consider a sequence of repeated interactions between an agent and an environment. Uncertainty about the environment is captured by a probability distribution over a space of hypotheses, which includes all computable functions. Given a utility function, we can evaluate the expected utility of any computational policy for interaction with the environment. After making some plausible assumptions (and maybe one not-so-plausible assumption), we show that if the utility function is unbounded, then the expected utility of any policy is undefined.

## 2. Ai Formalism

We will assume that the interaction between the agent and the environment takes place in discrete time-steps, or cycles. In cycle n**, the agent outputs an action**
yn ∈ Y , and the environment inputs to the agent a perception xn ∈ X. Y and X
are the sets of possible actions and perceptions, respectively, and are considered as subsets of N**. Thus the story of all interaction between agent and environment** is captured by the two sequences x = (x1, x2, . . .) and y = (y1, y2**, . . .**).

Let us introduce a notation for substrings. If s **is a sequence or string, and**
{a, b} ⊆ N, a ≤ b**, then define** s b a = (sa, sa+1**, . . . s**b).

We will denote the function instantiated by the environment as Q : Y
∗ → X,
so that ∀n ∈ N, xn = Q(y n 1**). This means that the perception generated by the**
environment at any given cycle is determined by the agent's actions on that and all previous cycles.

A policy for the agent is a function p : (Y
∗ × X∗) → Y **, so that an agent**
implementing p at time n **will choose an action** yn = py n−1 1, xn−1 1.

If, at any time, an agent adopts some policy p**, and continues to follow that**
policy forever, then p and Q **taken together completely determine the future of the**
sequences (xn) and (yn**). We are particularly interested in the future sequence of**
perceptions, so we will define a *future function* Ψ (**Q, p, y**n 1
, xn 1
) = x∞
n+1.

Because the precise nature of the environment Q **is unknown to the agent, we**
will let Ω be the set of possible environments. Let F be a σ**-algebra on Ω, and** P : F → [0, 1] be a probability measure on F **which represents the agent's prior** information about the environment.

We will also define a function Γq : Y
∗ → X∗ **which represents the perception**
string output by environment q **given some action string. Let**
Γq (s) = qs 1 1
, q s 2 1
**, . . . , q** (s).

The agent will compare the quality of different outcomes using a utility **function**
U : XN → R**. We can then judge a policy by calculating the expected utility of the**
outcome given that policy, which can be written as

## (1) E (U (X N 1 Ψ (**Q, P, Y**N 1, Xn 1))|Γq(Y N 1) = X N 1)

...where Q is being treated as a random variable. When we write a string next to a sequence, as in x n 1Ψ (**Q, p, y**n 1
, xn 1
), we mean to concatenate them. Here, x n 1 represents what the agent has seen in the past, and Ψ (**Q, p, y**n 1
, xn 1
) represents something the agent may see in the future. By concatenating them, we get a complete sequence of perceptions, which is the input required by the utility function U.

Notice that the expected utility above is a conditional expectation. **Except**
on the very first time-step, the agent will already have some knowledge about