2 PETER DE BLANC THE SINGULARITY INSTITUTE FOR ARTIFICIAL INTELLIGENCE
the values in its reward channel. Humans and other animals partially rely on a similar sort of reinforcement learning, but not entirely so.

We find the reinforcement learning approach unsatisfactory. As builders of artificial agents, we care about the changes to the environment that the agent will effect; any reward signal that the agent processes is only a proxy for these external changes. We would like to encode this information directly into the agent's utility function, rather than in an external system that the agent may seek to manipulate.

## 3. Our Approach

We will approach this problem from the perspective of concrete, comprehensible ontologies. An AI programmer may specify an ontology by hand, and then specify a utility function for that ontology. We will then try to devise a systematic way to translate this utility function to different ontologies.

When using this method in practice, we might expect the agent to have a probability distribution over many ontologies, perhaps specified concisely by the programmer as members of a parametric family. The programmer would specify a utility function on some concrete ontology which would be automatically translated to all other ontologies before the agent is turned on. In this way the agent has a complete utility function.

However, for the purposes of this discussion, we may imagine that the agent has only two ontologies, one old and one new, which we may call O0 and O1. The agent's utility function is defined in terms of states of O0, but it now believes O1 to be a more accurate model of its environment. The agent now faces an ontological crisis - the problem of translating its utility function to the new ontology O1.

In this paper we will present a method for addressing these problems. Our intention, however, is not to close the book on ontological crises, but rather to open it. Our method is of an ad-hoc character and only defined for a certain class of ontologies. Furthermore it is not computationally tractable for large ontologies.

We hope that this discussion will inspire other thinkers to consider the problem of ontological crises and develop new solutions.

## 4. Finite State Models

We wil nowl consider a specific kind of ontology, which we may call a finite state model. These models have some finite set of possible hidden states, which the agent does not directly observe. On each time step, the model inputs some symbol (the agent's output), enters some hidden state, and outputs some symbol (the agent's input). The model's output depends (stochastically) only on its current state, while its state depends (stochastically) on both the input and the previous state.

![0_image_0.png](0_image_0.png)

