Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Is Batch Norm unique? An empirical investigation and prescription to
  emulate the best properties of common normalizers without batch dependence   We perform an extensive empirical study of the statistical properties of
Batch Norm and other common normalizers. This includes an examination of the
correlation between representations of minibatches, gradient norms, and Hessian
spectra both at initialization and over the course of training. Through this
analysis, we identify several statistical properties which appear linked to
Batch Norm's superior performance. We propose two simple normalizers,
PreLayerNorm and RegNorm, which better match these desirable properties without
involving operations along the batch dimension. We show that PreLayerNorm and
RegNorm achieve much of the performance of Batch Norm without requiring batch
dependence, that they reliably outperform LayerNorm, and that they can be
applied in situations where Batch Norm is ineffective.

11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
10
The categories of this paper are: q-bio
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
4
The categories of this paper are: cond-mat
28
The categories of this paper are: alg-geom
26
The categories of this paper are: bayes-an
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
33
The categories of this paper are: plasm-ph
27
The categories of this paper are: comp-gas
25
The categories of this paper are: patt-sol
2
The categories of this paper are: math
20
The categories of this paper are: acc-phys
5
The categories of this paper are: gr-qc