Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Finite Time Analysis of Linear Two-timescale Stochastic Approximation
  with Markovian Noise   Linear two-timescale stochastic approximation (SA) scheme is an important
class of algorithms which has become popular in reinforcement learning (RL),
particularly for the policy evaluation problem. Recently, a number of works
have been devoted to establishing the finite time analysis of the scheme,
especially under the Markovian (non-i.i.d.) noise settings that are ubiquitous
in practice. In this paper, we provide a finite-time analysis for linear two
timescale SA. Our bounds show that there is no discrepancy in the convergence
rate between Markovian and martingale noise, only the constants are affected by
the mixing time of the Markov chain. With an appropriate step size schedule,
the transient term in the expected error bound is $o(1/k^c)$ and the
steady-state term is ${\cal O}(1/k)$, where $c>1$ and $k$ is the iteration
number. Furthermore, we present an asymptotic expansion of the expected error
with a matching lower bound of $\Omega(1/k)$. A simple numerical experiment is
presented to support our theory.

31
The categories of this paper are: ao-sci
5
The categories of this paper are: gr-qc
35
The categories of this paper are: mtrl-th
19
The categories of this paper are: eess
37
The categories of this paper are: supr-con
29
The categories of this paper are: funct-an
8
The categories of this paper are: hep-ex
26
The categories of this paper are: bayes-an
30
The categories of this paper are: q-alg
9
The categories of this paper are: nlin
25
The categories of this paper are: patt-sol
12
The categories of this paper are: nucl-th
4
The categories of this paper are: cond-mat
33
The categories of this paper are: plasm-ph
11
The categories of this paper are: hep-lat
15
The categories of this paper are: nucl-ex