Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Transformers without Tears: Improving the Normalization of
  Self-Attention   We evaluate three simple, normalization-centric changes to improve
Transformer training. First, we show that pre-norm residual connections
(PreNorm) and smaller initializations enable warmup-free, validation-based
training with large learning rates. Second, we propose $\ell_2$ normalization
with a single scale parameter (ScaleNorm) for faster training and better
performance. Finally, we reaffirm the effectiveness of normalizing word
embeddings to a fixed length (FixNorm). On five low-resource translation pairs
from TED Talks-based corpora, these changes always converge, giving an average
+1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on
IWSLT'15 English-Vietnamese. We observe sharper performance curves, more
consistent gradient norms, and a linear relationship between activation scaling
and decoder depth. Surprisingly, in the high-resource setting (WMT'14
English-German), ScaleNorm and FixNorm remain competitive but PreNorm degrades
performance.

26
The categories of this paper are: bayes-an
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
29
The categories of this paper are: funct-an
35
The categories of this paper are: mtrl-th
8
The categories of this paper are: hep-ex
31
The categories of this paper are: ao-sci
12
The categories of this paper are: nucl-th
25
The categories of this paper are: patt-sol
32
The categories of this paper are: atom-ph
20
The categories of this paper are: acc-phys
21
The categories of this paper are: adap-org
33
The categories of this paper are: plasm-ph
30
The categories of this paper are: q-alg
36
The categories of this paper are: cmp-lg
6
The categories of this paper are: astro-ph