6 PETER DE BLANC THE SINGULARITY INSTITUTE FOR ARTIFICIAL INTELLIGENCE

![0_image_0.png](0_image_0.png)

The black arrows indicate near-certainty; the gray arrows indicate probabilities of about 12
.

If we compose φ with our utility function, we obtain a utility of 1 for the right square and a utility of 0 for the other squares, which agrees with our intuitions.

## 7. Outlook

Those wishing to extend our algorithm as presented may consider what to do when the agent's sensors or motors are replaced, how to deal with differently-sized time steps, how to deal with continuous models, and how to efficiently find mappings between larger, structured ontologies.

Furthermore, there remain difficult philosophical problems. We have made a distinction between the agent's uncertainty about which model is correct and the agent's uncertainty about which state the world is in within the model. We may wish to eliminate this distinction; we could specify a single model, but only give utilities for some states of the model. We would then like the agent to generalize this utility function to the entire state space of the model.

Human beings also confront ontological crises. We should find out what cognitive algorithms humans use to solve the same problems described in this paper. If we wish to build agents that maximize human values, this may be aided by knowing how humans re-interpret their values in new ontologies.

We hope that other thinkers will consider these questions carefully.

```
                    Acknowledgements
                                    
 Thanks to Roko Mijic, with whom I discussed these ideas in 2009, and to Steve
Rayhawk, who gave extensive criticism on earlier versions of this paper.

```

## References

[1] Hutter, M., *A Gentle Introduction to the Universal Algorithmic Agent AIXI*, Artificial General Intelligence (2006), Springer