Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Importance of Consistency in Training Deep Neural Networks   We explain that the difficulties of training deep neural networks come from a
syndrome of three consistency issues. This paper describes our efforts in their
analysis and treatment. The first issue is the training speed inconsistency in
different layers. We propose to address it with an intuitive,
simple-to-implement, low footprint second-order method. The second issue is the
scale inconsistency between the layer inputs and the layer residuals. We
explain how second-order information provides favorable convenience in removing
this roadblock. The third and most challenging issue is the inconsistency in
residual propagation. Based on the fundamental theorem of linear algebra, we
provide a mathematical characterization of the famous vanishing gradient
problem. Thus, an important design principle for future optimization and neural
network design is derived. We conclude this paper with the construction of a
novel contractive neural network.

14
The categories of this paper are: math-ph
35
The categories of this paper are: mtrl-th
13
The categories of this paper are: quant-ph
16
The categories of this paper are: stat
11
The categories of this paper are: hep-lat
30
The categories of this paper are: q-alg
15
The categories of this paper are: nucl-ex
25
The categories of this paper are: patt-sol
24
The categories of this paper are: dg-ga
0
The categories of this paper are: hep-ph
9
The categories of this paper are: nlin
26
The categories of this paper are: bayes-an
33
The categories of this paper are: plasm-ph
4
The categories of this paper are: cond-mat
18
The categories of this paper are: econ
5
The categories of this paper are: gr-qc