Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Multi-step Off-policy Learning Without Importance Sampling Ratios   To estimate the value functions of policies from exploratory data, most
model-free off-policy algorithms rely on importance sampling, where the use of
importance sampling ratios often leads to estimates with severe variance. It is
thus desirable to learn off-policy without using the ratios. However, such an
algorithm does not exist for multi-step learning with function approximation.
In this paper, we introduce the first such algorithm based on
temporal-difference (TD) learning updates. We show that an explicit use of
importance sampling ratios can be eliminated by varying the amount of
bootstrapping in TD updates in an action-dependent manner. Our new algorithm
achieves stability using a two-timescale gradient-based TD update. A prior
algorithm based on lookup table representation called Tree Backup can also be
retrieved using action-dependent bootstrapping, becoming a special case of our
algorithm. In two challenging off-policy tasks, we demonstrate that our
algorithm is stable, effectively avoids the large variance issue, and can
perform substantially better than its state-of-the-art counterpart.

0
The categories of this paper are: hep-ph
32
The categories of this paper are: atom-ph
11
The categories of this paper are: hep-lat
9
The categories of this paper are: nlin
5
The categories of this paper are: gr-qc
12
The categories of this paper are: nucl-th
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
15
The categories of this paper are: nucl-ex
10
The categories of this paper are: q-bio
7
The categories of this paper are: hep-th
33
The categories of this paper are: plasm-ph
23
The categories of this paper are: solv-int
4
The categories of this paper are: cond-mat
6
The categories of this paper are: astro-ph
2
The categories of this paper are: math