Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Stochastic Gradient Descent: Going As Fast As Possible But Not Faster   When applied to training deep neural networks, stochastic gradient descent
(SGD) often incurs steady progression phases, interrupted by catastrophic
episodes in which loss and gradient norm explode. A possible mitigation of such
events is to slow down the learning process. This paper presents a novel
approach to control the SGD learning rate, that uses two statistical tests. The
first one, aimed at fast learning, compares the momentum of the normalized
gradient vectors to that of random unit vectors and accordingly gracefully
increases or decreases the learning rate. The second one is a change point
detection test, aimed at the detection of catastrophic learning episodes; upon
its triggering the learning rate is instantly halved. Both abilities of
speeding up and slowing down the learning rate allows the proposed approach,
called SALeRA, to learn as fast as possible but not faster. Experiments on
standard benchmarks show that SALeRA performs well in practice, and compares
favorably to the state of the art.

34
The categories of this paper are: chem-ph
12
The categories of this paper are: nucl-th
17
The categories of this paper are: q-fin
32
The categories of this paper are: atom-ph
6
The categories of this paper are: astro-ph
5
The categories of this paper are: gr-qc
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
3
The categories of this paper are: physics
22
The categories of this paper are: chao-dyn
19
The categories of this paper are: eess
13
The categories of this paper are: quant-ph
10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
25
The categories of this paper are: patt-sol
2
The categories of this paper are: math