Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Infinite-Horizon Policy-Gradient Estimation   Gradient-based approaches to direct policy search in reinforcement learning
have received much recent attention as a means to solve problems of partial
observability and to avoid some of the problems associated with policy
degradation in value-function methods. In this paper we introduce GPOMDP, a
simulation-based algorithm for generating a {\em biased} estimate of the
gradient of the {\em average reward} in Partially Observable Markov Decision
Processes (POMDPs) controlled by parameterized stochastic policies. A similar
algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The
algorithm's chief advantages are that it requires storage of only twice the
number of policy parameters, uses one free parameter $\beta\in [0,1)$ (which
has a natural interpretation in terms of bias-variance trade-off), and requires
no knowledge of the underlying state. We prove convergence of GPOMDP, and show
how the correct choice of the parameter $\beta$ is related to the {\em mixing
time} of the controlled POMDP. We briefly describe extensions of GPOMDP to
controlled Markov chains, continuous state, observation and control spaces,
multiple-agents, higher-order derivatives, and a version for training
stochastic policies with internal states. In a companion paper (Baxter,
Bartlett, & Weaver, 2001) we show how the gradient estimates generated by
GPOMDP can be used in both a traditional stochastic gradient algorithm and a
conjugate-gradient procedure to find local optima of the average reward

32
The categories of this paper are: atom-ph
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
35
The categories of this paper are: mtrl-th
31
The categories of this paper are: ao-sci
27
The categories of this paper are: comp-gas
24
The categories of this paper are: dg-ga
26
The categories of this paper are: bayes-an
15
The categories of this paper are: nucl-ex
4
The categories of this paper are: cond-mat
12
The categories of this paper are: nucl-th
21
The categories of this paper are: adap-org
19
The categories of this paper are: eess
28
The categories of this paper are: alg-geom
0
The categories of this paper are: hep-ph
6
The categories of this paper are: astro-ph