Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Improve Generalization and Robustness of Neural Networks via Weight
  Scale Shifting Invariant Regularizations   Using weight decay to penalize the L2 norms of weights in neural networks has
been a standard training practice to regularize the complexity of networks. In
this paper, we show that a family of regularizers, including weight decay, is
ineffective at penalizing the intrinsic norms of weights for networks with
positively homogeneous activation functions, such as linear, ReLU and
max-pooling functions. As a result of homogeneity, functions specified by the
networks are invariant to the shifting of weight scales between layers. The
ineffective regularizers are sensitive to such shifting and thus poorly
regularize the model capacity, leading to overfitting. To address this
shortcoming, we propose an improved regularizer that is invariant to weight
scale shifting and thus effectively constrains the intrinsic norm of a neural
network. The derived regularizer is an upper bound for the input gradient of
the network so minimizing the improved regularizer also benefits the
adversarial robustness. Residual connections are also considered and we show
that our regularizer also forms an upper bound to input gradients of such a
residual network. We demonstrate the efficacy of our proposed regularizer on
various datasets and neural network architectures at improving generalization
and adversarial robustness.

4
The categories of this paper are: cond-mat
11
The categories of this paper are: hep-lat
33
The categories of this paper are: plasm-ph
36
The categories of this paper are: cmp-lg
6
The categories of this paper are: astro-ph
18
The categories of this paper are: econ
17
The categories of this paper are: q-fin
32
The categories of this paper are: atom-ph
34
The categories of this paper are: chem-ph
25
The categories of this paper are: patt-sol
22
The categories of this paper are: chao-dyn
12
The categories of this paper are: nucl-th
13
The categories of this paper are: quant-ph
35
The categories of this paper are: mtrl-th
10
The categories of this paper are: q-bio
28
The categories of this paper are: alg-geom