

![0_image_0.png](0_image_0.png)

of the probabilities conditioned on the state variable, even though the state space can be of high dimension.

## 3.1.4 P Olicy H I E R C H Y

Using abstract policies as the building blocks, we can construct a hierarchy of abstract policies as follows:
Definition 3 (Policy hierarchy). A policy hierarchy is a sequence H = (Π 0 , Π 1 , . . , Π K )
where K is the number of levels in the hierarchy, Π 0 is a set of primitive actions, and for k = 1,..., K, I k is a set of abstract policies over the policies in I k -1.

When a top-level policy π K is executed, it invokes a sequence of level-(K-1) policies, each of which invokes a sequence of level-(K-2) policies and so on. A level-1 policy will invoke a sequence of primitive actions which leads to a sequence of states. Thus, the execution of π K generates an overall state sequence ( so, s 1, . . , st, . . .) that terminates in one of the destination states in D n K . When K = 1 this sequence is simply a Markov chain (with suitable stopping conditions). However, for K ≥ 2, it will generally be non-Markovian, despite the fact that all the policies are Markov, i.e., they select the lower level policies based solely on the current state (Sutton et al., 1999). This is because knowing the current state s t alone does not provide information about the current intermediate-level policies, which can affect the selection of the next state s t+1 . Intuitively, this means that an agent's behaviour to achieve a given goal is usually non-Markovian, since its choice of actions depends not only on the current state, but also on the current intermediate intentions of the agent.

We term the dynamical process in executing a top-level abstract policy π K the Abstract Markov Model (AMM). When the states are only partially observable, the observation can be modelled by the usual observation model Pr( o t | s t ) = ω ( s t , o t ). The resulting process is termed the Abstract Hidden Markov Model (AHMM) since the states are hidden as in the Hidden Markov Model (Rabiner, 1989).

The idea of having a higher level policy controlling the lower level ones in an MDP
can be traced back to the work by Forestier and Varaiya (1978), who investigated a two layer structure similar to our 2-level policy hierarchy with deterministic stopping condition.

Forestier and Varaiya showed that that the sub-process, obtained by sub-sampling the state