Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic
  Objectives with Skewed Hessian Spectrums   Learning rate schedulers have been widely adopted in training deep neural
networks. Despite their practical importance, there is a discrepancy between
its practice and its theoretical analysis. For instance, it is not known what
schedules of SGD achieve best convergence, even for simple problems such as
optimizing quadratic objectives. So far, step decay has been one of the
strongest candidates under this setup, which is proved to be nearly optimal
with a $\mathcal{O}(\log T)$ gap. However, according to our analysis, this gap
turns out to be $\Omega(\log T)$ in a wide range of settings, which throws the
schedule optimality problem into an open question again. Towards answering this
reopened question, in this paper, we propose Eigencurve, the first family of
learning rate schedules that can achieve minimax optimal convergence rates (up
to a constant) for SGD on quadratic objectives when the eigenvalue distribution
of the underlying Hessian matrix is skewed. The condition is quite common in
practice. Experimental results show that Eigencurve can significantly
outperform step decay in image classification tasks on CIFAR-10, especially
when the number of epochs is small. Moreover, the theory inspires two simple
learning rate schedulers for practical applications that can approximate
Eigencurve. For some problems, the optimal shape of the proposed schedulers
resembles that of cosine decay, which sheds light to the success of cosine
decay for such situations. For other situations, the proposed schedulers are
superior to cosine decay.

11
The categories of this paper are: hep-lat
32
The categories of this paper are: atom-ph
27
The categories of this paper are: comp-gas
37
The categories of this paper are: supr-con
31
The categories of this paper are: ao-sci
25
The categories of this paper are: patt-sol
21
The categories of this paper are: adap-org
9
The categories of this paper are: nlin
6
The categories of this paper are: astro-ph
24
The categories of this paper are: dg-ga
35
The categories of this paper are: mtrl-th
3
The categories of this paper are: physics
14
The categories of this paper are: math-ph
15
The categories of this paper are: nucl-ex
13
The categories of this paper are: quant-ph
10
The categories of this paper are: q-bio