Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Simple and optimal methods for stochastic variational inequalities, II:
  Markovian noise and policy evaluation in reinforcement learning   The focus of this paper is on stochastic variational inequalities (VI) under
Markovian noise. A prominent application of our algorithmic developments is the
stochastic policy evaluation problem in reinforcement learning. Prior
investigations in the literature focused on temporal difference (TD) learning
by employing nonsmooth finite time analysis motivated by stochastic subgradient
descent leading to certain limitations. These encompass the requirement of
analyzing a modified TD algorithm that involves projection to an a-priori
defined Euclidean ball, achieving a non-optimal convergence rate and no clear
way of deriving the beneficial effects of parallel implementation. Our approach
remedies these shortcomings in the broader context of stochastic VIs and in
particular when it comes to stochastic policy evaluation. We developed a
variety of simple TD learning type algorithms motivated by its original version
that maintain its simplicity, while offering distinct advantages from a
non-asymptotic analysis point of view. We first provide an improved analysis of
the standard TD algorithm that can benefit from parallel implementation. Then
we present versions of a conditional TD algorithm (CTD), that involves periodic
updates of the stochastic iterates, which reduce the bias and therefore exhibit
improved iteration complexity. This brings us to the fast TD (FTD) algorithm
which combines elements of CTD and the stochastic operator extrapolation method
of the companion paper. For a novel index resetting policy FTD exhibits the
best known convergence rate. We also devised a robust version of the algorithm
that is particularly suitable for discounting factors close to 1.

22
The categories of this paper are: chao-dyn
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
25
The categories of this paper are: patt-sol
31
The categories of this paper are: ao-sci
15
The categories of this paper are: nucl-ex
32
The categories of this paper are: atom-ph
16
The categories of this paper are: stat
35
The categories of this paper are: mtrl-th
4
The categories of this paper are: cond-mat
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
24
The categories of this paper are: dg-ga
37
The categories of this paper are: supr-con
34
The categories of this paper are: chem-ph
14
The categories of this paper are: math-ph