Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Towards Deeper Deep Reinforcement Learning with Spectral Normalization   In computer vision and natural language processing, innovations in model
architecture that increase model capacity have reliably translated into gains
in performance. In stark contrast with this trend, state-of-the-art
reinforcement learning (RL) algorithms often use small MLPs, and gains in
performance typically originate from algorithmic innovations. It is natural to
hypothesize that small datasets in RL necessitate simple models to avoid
overfitting; however, this hypothesis is untested. In this paper we investigate
how RL agents are affected by exchanging the small MLPs with larger modern
networks with skip connections and normalization, focusing specifically on
actor-critic algorithms. We empirically verify that naively adopting such
architectures leads to instabilities and poor performance, likely contributing
to the popularity of simple models in practice. However, we show that dataset
size is not the limiting factor, and instead argue that instability from taking
gradients through the critic is the culprit. We demonstrate that spectral
normalization (SN) can mitigate this issue and enable stable training with
large modern architectures. After smoothing with SN, larger models yield
significant performance improvements -- suggesting that more "easy" gains may
be had by focusing on model architectures in addition to algorithmic
innovations.

9
The categories of this paper are: nlin
26
The categories of this paper are: bayes-an
8
The categories of this paper are: hep-ex
21
The categories of this paper are: adap-org
31
The categories of this paper are: ao-sci
33
The categories of this paper are: plasm-ph
35
The categories of this paper are: mtrl-th
5
The categories of this paper are: gr-qc
19
The categories of this paper are: eess
2
The categories of this paper are: math
4
The categories of this paper are: cond-mat
28
The categories of this paper are: alg-geom
32
The categories of this paper are: atom-ph
17
The categories of this paper are: q-fin
22
The categories of this paper are: chao-dyn
16
The categories of this paper are: stat