Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy   We study the global convergence and global optimality of actor-critic, one of
the most popular families of reinforcement learning algorithms. While most
existing works on actor-critic employ bi-level or two-timescale updates, we
focus on the more practical single-timescale setting, where the actor and
critic are updated simultaneously. Specifically, in each iteration, the critic
update is obtained by applying the Bellman evaluation operator only once while
the actor is updated in the policy gradient direction computed using the
critic. Moreover, we consider two function approximation settings where both
the actor and critic are represented by linear or deep neural networks. For
both cases, we prove that the actor sequence converges to a globally optimal
policy at a sublinear $O(K^{-1/2})$ rate, where $K$ is the number of
iterations. To the best of our knowledge, we establish the rate of convergence
and global optimality of single-timescale actor-critic with linear function
approximation for the first time. Moreover, under the broader scope of policy
optimization with nonlinear function approximation, we prove that actor-critic
with deep neural network finds the globally optimal policy at a sublinear rate
for the first time.

0
The categories of this paper are: hep-ph
22
The categories of this paper are: chao-dyn
18
The categories of this paper are: econ
33
The categories of this paper are: plasm-ph
32
The categories of this paper are: atom-ph
14
The categories of this paper are: math-ph
9
The categories of this paper are: nlin
17
The categories of this paper are: q-fin
24
The categories of this paper are: dg-ga
7
The categories of this paper are: hep-th
27
The categories of this paper are: comp-gas
10
The categories of this paper are: q-bio
37
The categories of this paper are: supr-con
21
The categories of this paper are: adap-org
35
The categories of this paper are: mtrl-th
13
The categories of this paper are: quant-ph