Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Train Feedfoward Neural Network with Layer-wise Adaptive Rate via
  Approximating Back-matching Propagation   Stochastic gradient descent (SGD) has achieved great success in training deep
neural network, where the gradient is computed through back-propagation.
However, the back-propagated values of different layers vary dramatically. This
inconsistence of gradient magnitude across different layers renders
optimization of deep neural network with a single learning rate problematic. We
introduce the back-matching propagation which computes the backward values on
the layer's parameter and the input by matching backward values on the layer's
output. This leads to solving a bunch of least-squares problems, which requires
high computational cost. We then reduce the back-matching propagation with
approximations and propose an algorithm that turns to be the regular SGD with a
layer-wise adaptive learning rate strategy. This allows an easy implementation
of our algorithm in current machine learning frameworks equipped with
auto-differentiation. We apply our algorithm in training modern deep neural
networks and achieve favorable results over SGD.

9
The categories of this paper are: nlin
37
The categories of this paper are: supr-con
10
The categories of this paper are: q-bio
18
The categories of this paper are: econ
29
The categories of this paper are: funct-an
36
The categories of this paper are: cmp-lg
23
The categories of this paper are: solv-int
31
The categories of this paper are: ao-sci
24
The categories of this paper are: dg-ga
0
The categories of this paper are: hep-ph
33
The categories of this paper are: plasm-ph
11
The categories of this paper are: hep-lat
19
The categories of this paper are: eess
5
The categories of this paper are: gr-qc
14
The categories of this paper are: math-ph
27
The categories of this paper are: comp-gas