replaced by

$$\frac{\nabla\frac{d\mu(\theta,Y_{t})}{d\lambda}(U_{t})}{\frac{d\mu(\theta,Y_{t})}{d\lambda}(U_{t})}.$$

Under Assumptions 1, 2 and 6, this algorithm, starting from any initial state X0 will generate a sequence ∆0, ∆1, . . . , ∆t**, . . .** *satisfying*

$$\operatorname*{lim}_{t\to\infty}\Delta_{t}=\nabla_{\beta}\eta\quad\mathrm{w.p.1.}$$
$\dim_X$ B. 
Proof. See Appendix B

## 7. New Results

Since the first version of this paper, we have extended **GPOMDP** to several new settings, and also proved some new properties of the algorithm. In this section we briefly outline these results.

## 7.1 Multiple Agents

Instead of a single agent generating actions according to µ(**θ, y**), suppose we have multiple agents i = 1**, . . . , n**a, each with their own parameter set θ iand distinct observation of the environment y i, and that generate their own actions u iaccording to a policy µui (θ i, yi). If the agents all receive the same reward signal r(Xt) (they may be cooperating to solve the same task, for example),
then **GPOMDP** can be applied to the collective **POMDP** obtained by concatenating the observations, controls, and parameters into single vectors y =-y 1**, . . . , y**na, u =-u 1**, . . . , u**na, and θ =-θ 1**, . . . , θ**narespectively. An easy calculation shows that the gradient estimate ∆ generated by **GPOMDP** in the collective case is precisely the same as that obtained by applying **GPOMDP** to each agent independently, and then concatenating the results. That is, ∆ = -∆1**, . . . ,** ∆na, where
∆iis the estimate produced by **GPOMDP** applied to agent i. This leads to an on-line algorithm in which the agents adjust their parameters independently and without any explicit communication, yet collectively the adjustments are maximizing the global average reward. For similar observations in the context of **REINFORCE** and **VAPS**, see Peshkin et al. (2000). This algorithm gives a biologically plausible synaptic weight-update rule when applied to networks of spiking neurons in which the neurons are regarded as independent agents (Bartlett & Baxter, 1999), and has shown some promise in a network routing application (Tao, Baxter, & Weaver, 2001).

## 7.2 Policies With Internal States

So far we have only considered purely reactive or *memoryless* policies in which the chosen control is a function of only the current observation. **GPOMDP** is easily extended to cover the case of policies that depend on finite histories of observations Yt, Yt−1**, . . . , Y**t−k, but in general, for *optimal* control of **POMDP**s, the policy must be a function of the *entire* observation history. Fortunately, the observation history may be summarized in the form of a *belief state* (the current distribution over states), which is itself updated based only upon the current observation, and knowledge of which is sufficient for optimal behaviour (Smallwood & Sondik, 1973; Sondik, 1978). An extension of GPOMDP to policies with parameterized internal belief states is described by Aberdeen and Baxter (2001), similar in spirit to the extension of **VAPS** and **REINFORCE** described by Meuleau et al. (1999).