
## 6.3 Extensions To Infinite State, Observation, And Control Spaces

The convergence proof for Algorithm 2 relied on finite state (S), observation (Y) and control (U)
spaces. However, it should be clear that with no modification Algorithm 2 can be applied immediately to **POMDP**s with countably or uncountably infinite S and Y, and countable U. All that changes is that pij (u) becomes a *kernel* p(x, x′, u) and ν(i) becomes a density on observations. In addition, with the appropriate interpretation of ∇µ/µ, it can be applied to uncountable U. Specifically, if U is a subset of R
N then µ(**y, θ**) will be a probability *density* function on U with µu(**y, θ**)
the density at u. If U and Y are subsets of Euclidean space (but S is a finite set), Theorem 5 can be extended to show that the estimates produced by this algorithm converge almost surely to ∇βη. In fact, we can prove a more general result that implies both this case of densities on subsets of R
N as well as the finite case of Theorem 5. We allow U and Y to be general spaces satisfying the following topological assumption. (For definitions see, for example, (Dudley, 1989).)
Assumption 5. The control space U has an associated topology that is separable, Hausdorff, and first-countable. For the corresponding Borel σ-algebra B *generated by this topology, there is a* σ-finite measure λ defined on the measurable space (U,B). We say that λ *is the* reference measure for U.

Similarly, the observation space Y has a topology, Borel σ*-algebra, and reference measure* satisfying the same conditions.

In the case of Theorem 5, where U and Y are finite, the associated reference measure is the counting measure. For U = R
N and Y = RM, the reference measure is Lebesgue measure. We assume that the distributions ν(i) and µ(**θ, y**) are absolutely continuous with respect to the reference measures, and the corresponding Radon-Nikodym derivatives (probability masses in the finite case, densities in the Euclidean case) satisfy the following assumption.

Assumption 6. *For every* y ∈ Y and θ ∈ R
K*, the probability measure* µ(**θ, y**) is absolutely contin-

uous with respect to the reference measure for U*. For every* i ∈ S*, the probability measure* ν(i) is
absolutely continuous with respect to the reference measure for Y.
Let λ be the reference measure for U*. For all* u ∈ U, y ∈ Y, θ ∈ R
K*, and* k ∈ {1**, . . . , K**}*, the*
derivatives∂
 A 2-bit 2 < 20, 3 < 1  $\dfrac{\partial}{\partial\theta_k}\dfrac{d\mu(\theta,y)}{d\lambda}(u)$  $\mid\ \alpha\ \ d\mu_1(\theta,y)=-1$
exist and the ratios

![0_image_0.png](0_image_0.png)

are bounded by Bµ < ∞.
With these assumptions, we can replace µ in Algorithm 2 with the Radon-Nikodym derivative
of µ with respect to the reference measure on U. In this case, we have the following convergence result. This generalizes Theorem 5, and also applies to densities µ on a Euclidean space U.
Theorem 6. Suppose the control space U and the observation space Y *satisfy Assumption 5 and let* λ be the reference measure on the control space U*. Consider Algorithm 2 with*
$$\frac{\nabla\mu_{U_{t}}(\theta,Y_{t})}{\mu_{U_{t}}(\theta,Y_{t})}$$