Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Blending MPC & Value Function Approximation for Efficient Reinforcement
  Learning   Model-Predictive Control (MPC) is a powerful tool for controlling complex,
real-world systems that uses a model to make predictions about future behavior.
For each state encountered, MPC solves an online optimization problem to choose
a control action that will minimize future cost. This is a surprisingly
effective strategy, but real-time performance requirements warrant the use of
simple models. If the model is not sufficiently accurate, then the resulting
controller can be biased, limiting performance. We present a framework for
improving on MPC with model-free reinforcement learning (RL). The key insight
is to view MPC as constructing a series of local Q-function approximations. We
show that by using a parameter $\lambda$, similar to the trace decay parameter
in TD($\lambda$), we can systematically trade-off learned value estimates
against the local Q-function approximations. We present a theoretical analysis
that shows how error from inaccurate models in MPC and value function
estimation in RL can be balanced. We further propose an algorithm that changes
$\lambda$ over time to reduce the dependence on MPC as our estimates of the
value function improve, and test the efficacy our approach on challenging
high-dimensional manipulation tasks with biased models in simulation. We
demonstrate that our approach can obtain performance comparable with MPC with
access to true dynamics even under severe model bias and is more sample
efficient as compared to model-free RL.

22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
5
The categories of this paper are: gr-qc
29
The categories of this paper are: funct-an
31
The categories of this paper are: ao-sci
0
The categories of this paper are: hep-ph
19
The categories of this paper are: eess
6
The categories of this paper are: astro-ph
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
2
The categories of this paper are: math
13
The categories of this paper are: quant-ph
12
The categories of this paper are: nucl-th
4
The categories of this paper are: cond-mat
18
The categories of this paper are: econ
30
The categories of this paper are: q-alg